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Executive Summary 


This report presents the results of a benchmark evaluation of a prototype design for an 
Interactive Electronic NATOPS (IE-NATOPS) to support Navy aircrews. The IE- 
NATOPS design was developed following an investigation of aircrew information 
requirements for decision making associated with monitoring of aircraft system health 
and management of in-flight mechanical malfunctions. It was determined in two 
simulator-based studies with H-46 aircrews that various types of aiding in support of 
mechanical (and other subsystem) health monitoring and problem diagnosis are both 
desirable and feasible, but that such information must be fully integrated into the Navy 
helicopter NATOPS flight manual and applicable checklists that represent training 
doctrine for aircraft operations and are used by aircrews during normal and emergency 
in-flight operations. The baseline concept for IE-NATOPS is to provide the pilot in the 
cockpit with a computer-based presentation of the information that is currently provided 
in hard-copy manuals, providing operating information and procedures that are specific to 
each Navy aircraft. For the Navy, as well as for virtually all other military and 
commercial aviation applications, the benefits are compelling - context sensitivity, fast 
and cheap “updatability”, and the capacity for embedded intelligence and embedded 
intelligent training. Initial research efforts in this program focused on the H-46 
helicopter, but later developments were for the SH-60F. Multiple options are under 
consideration for implementation of IE-NATOPS in the cockpit, including via an 
“electronic kneeboard” system that would interface with other aircraft information and 
alerting systems. 

The approach to design of IE-NATOPS involved the development of a design 
specification, production of a prototype demonstration, and the identification and 
evaluation of Navy institutional integration issues. The design specification focused on: 

• aircrew access during an emergency with automatic triggering, 

• aircrew access during an emergency based on aircrew initiative and 
problem identification, 

• aircrew access during non-emergency periods, and 

• aircrew access to automatic and interactive performance data 
calculations. 

The design specification was based on interviews with experienced Navy aircrews 
regarding the concept of an IE-NATOPS with specific reference to the Navy SH-60F 
helicopter. Based on those interviews and an analysis of aircrew information needs, a 
design specification was formulated addressing both event-triggered needs and aircrew 
initiative information needs. Six modes of IE-NATOPS functionality were defined: Pilot 
Checklists (PCL), NATOPS tutorial text and graphics, performance data, alerts, trends, 
and notes. Specific IE-NATOPS design features were derived from analyses of the 
current NATOPS organization to identify the Parts, Chapters, Sections, and sub-Sections 
that are used most by the aircrew in the cockpit, and to suggest potential design features 
for an effective interface structure. 

An experimental evaluation was performed to compare aircrew performance when using 
the traditional paper copy of NATOPS and the associated Pocket Check List (PCL) with 
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that of the IE-NATOPS system. This evaluation effort attempted to establish whether or 
not there are notable differences in resolving problem situations dependent upon the 
format of the NATOPS and PCL used. Major findings of this study demonstrated that the 
average response time for training scenarios using the electronic version of NATOPS was 
significantly faster than the average response time using the paper version. Since the 
training scenarios focused on pure access time to specific information, this is indicative 
of a strong advantage of EE-NATOPS over the paper version when explicit instructions 
are given as to the location of specific NATOPS information. In other experimental 
trials, more self-directed problem solving and decision-making were required to 
determine the path to the required information and, not surprisingly, the comparison of 
IE-NATOPS versus hard-copy performance varied considerably with problem 
characteristics. Overall, however, there were no significant differences in response time 
between electronic and paper versions of NATOPS for the self-directed scenarios. Given 
that the aviators who participated in this benchmark evaluation of the EE-NATOPS had 
less than an hour of training on IE-NATOPS, and given that the aviators were highly 
experienced in the use of the paper NATOPS and PCL for accessing information to 
resolve problem scenarios, the results from the problem solving trials are indicative of a 
system that may prove through further training and experience to be superior to the 
existing method of accessing hard-copy NATOPS data in the aircraft. 

Since several types of NATOPS information involves graphic presentation and since 
graphic displays can be awkward on the kind of small-screen devices envisioned for IE- 
NATOPS, several graphic interaction methodologies were formulated and investigated to 
support IE-NATOPS graphics requirements. Graphic interaction concepts that were 
investigated include an “electronic magnifying glass,” highlighting of axis lines and 
parameter curves, and dynamic labeling. Illustrative applications were made in the 
context of a typical aircraft performance chart as well as for a wind envelope chart. 

These alternative concepts for interacting with NATOPS charts were simply presented to 
the participating pilots and feedback was solicited on each of the concepts. In general, 
the results of the interviews suggested that pilots felt the techniques used when 
interacting with graphics are very important to the usefulness of the IE-NATOPS system. 
Due to the small size of the targeted IE-NATOPS display, pilots felt that the ways in 
which they interacted with graphics in the paper NATOPS manual would not be adequate 
when a performance chart was reduced in size. More specifically, every pilot felt that the 
best, and most useful, concept was dynamic labeling. They felt that this type of display 
provided them with all of the values that are of interest when they are interacting with a 
performance chart. Each pilot liked the fact that all important parameters are quickly 
available and felt that the feedback provided to them through the use of highlighting and 
labeling would allow them to immediately determine the particular value for which they 
were looking. However, some pilots felt that it was unnecessary to highlight the entire 
parameter curve. Some suggested that the value of the parameter simply be displayed off 
the tip of the stylus to limit the spread of information within the display. 

IE-NATOPS can offer several significant benefits over hard-copy checklists and flight 
manuals for the Navy helicopter aircrew. It can provide for rapid access to emergency 
procedures by the aircrew, both via automatic alerting (i.e., through an automatic 
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diagnostic system) and via manual search by the aircrew. It can also provide an efficient 
mechanism to facilitate Navy control of NATOPS revisions, since the electronic 
implementation will afford the opportunity for rapid, inexpensive dissemination of 
document revisions. In order to achieve these benefits, it is necessary to insure that the 
product is effectively usable in all relevant aspects (e.g., accessible to all aircrew, 
searchable with all relevant strategies, readable in all conditions, etc.). This effort has 
developed an interface design that illustrates how each of the access modes can be 
effectively supported. 

EE-NATOPS has potential uses not only as a decision-aid while in the cockpit, but also as 
a training tool that can be implemented in support of current NATOPS training 
procedures. Because the tool is electronic, it offers a portable training mode as well as a 
method to support delivery of training via the design of event-based scenarios in which 
specific competencies dealing with EE-NATOPS are targeted. This would allow targeted 
practice and feedback to be built into the system to augment the feedback given by the 
instructors). To pursue such training applications, a targeted analysis could be 
conducted of current NATOPS training procedures to: (a) identify strengths and 
weaknesses of the current method, (b) determine where IE-NATOPS may fit into the 
training rotation, and (c) identify the competencies that should be addressed with this 
tool. 

Further IE-NATOPS development efforts are envisioned in both operational development 
and research. Operational development must be conducted to resolve issues in the areas 
of institutionalization, aircraft integration, and interface design. Institutionalization 
issues will include coordination with NATOPS committee members and the Naval Air 
Systems Command program office for training systems (PMA 205) to identify concept 
issues related to training and to integration of DE-NATOPS into the NATOPS creation 
and update process. Flight evaluations of a certified EE-NATOPS system will be required 
to address aircraft integration and interface design issues. Future research efforts will be 
required to refine diagnosis logic for cockpit alerting and tools to interact with graphics 
on small screens. Research is also warranted to investigate alternative tools for 
performance data calculations and checklist interface features. 
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1. Introduction 


1.1 Background 

Navy aircraft pilots and aircrews are guided in their in-flight decision processes by the 
Naval Air Training and Operating Procedures Standards (NATOPS) manuals, which 
provide text descriptions of systems, tactics, responsibilities, and equipment usage 
protocols, as well as checklists and graphic aids for calculation of aircraft performance 
data. In the present project, an interactive electronic NATOPS (IE-NATOPS) has been 
developed to facilitate the in-flight decision-making processes requiring NATOPS data. 
By making human-centered improvements to the organization, presentation, and 
accessibility of the NATOPS data, it is anticipated that more timely resolution of systems 
and performance related problems occurring in flight may result than is observed with the 
current paper-based NATOPS format (Degani & Wiener, 1990). 

Before designing IE-NATOPS, aircrew information requirements in the area of aircraft 
health management were investigated through a series of two empirical studies with naval 
helicopter pilots. Both studies were conducted with H-46 aircrews at Naval Air Station 
North Island using a motion-base simulator in order to: 

• determine procedures currently used by aircrew in mitigating in-flight 
mechanical systems emergencies, 

• determine the decision processes employed by aircrew in resolving these 
emergencies, 

• gather aircrew evaluations of emerging mechanical fault diagnostic 
technologies, and 

• determine how information available from emerging mechanical 
diagnostic systems could best be displayed to aircrew to predict and 
mitigate these emergencies. 

The purpose of the first information requirements study was to conduct systematic 
interviews with flight crews regarding their current procedures for in-flight mechanical 
system emergencies (Glenn et al., 1998; Deaton et al., 1997 a & b). The study assessed 
cognitive activities expected to be important when responding to mechanical 
emergencies, and the attention devoted to each activity. A secondary goal of the first 
study was to identify the information requirements of a new diagnostic system to be used 
to predict and mitigate in-flight mechanical system emergencies. The study represented a 
first step towards identifying the information requirements of aircrew in determining the 
status of aircraft mechanical systems using advanced sensor and processing technology. 

Other research efforts in this program addressed the development of techniques and 
assessment of general technological feasibility for generating various kinds of 
information to aids for mechanical system health monitoring and diagnosis. Since these 
efforts are only tangentially relevant to the IE-NATOPS design that is the focus of this 
report, they will not be further discussed here; rather the interested reader is referred to 
Byington et al. (1999) for a general overview of the kinds of information envisioned, to 
Garga et al. (2001) for an approach to an automated reasoning technique for this purpose, 
and to Campbell et al. (2001) for a description of specific diagnostic algorithms based on 
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mechanical equipment vibration spectra. The experimental designs for both of our 
information requirements studies were based on these assessments and projects for the 
near-term feasibility of obtaining the various kinds of aiding information that were 
predicated in these studies. 

As reported in Glenn et al. (1998) and Deaton et al. (1997 a & b), the emphasis in the first 
study was to use survey techniques to assess aircrews' use of information, potential 
sources of workload, and the utility of diagnostic systems. A key finding of that study 
was the need for a system that could diagnose mechanical problems and assess the impact 
of those problems on a mission. Aircrews also requested a diagnostic system that could 
provide action recommendations to help complete a mission. Figure 1 presents the 
frequencies with which aircrews indicated interest in various categories of information 
during simulated emergencies. Results are included for both the questionnaire and from 
an analysis of actual aircrew communication during a simulated mission. 

Comparisons for Malfunction Sections 



Corroboration Diagnosis Prognosis Action Rec. Other 


Figure 1. Information Requirements Study 1 Results 


The second information requirements study was designed to assess the usefulness of 
various kinds of information relevant to mechanical fault management (see Byington et 
al., 1999). Accordingly, it explored the feasibility and potential benefit of new 
technologies to help aircrews in mechanical fault management. More specifically, the 
purpose of this study was twofold: 
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(1) to evaluate the value that various kinds of information have on the 
ability of the aircrew to successfully manage in-flight mechanical faults, and 

(2) to demonstrate an initial notational interface concept and explore its 
potential benefits to the aircrew. 

This second study built upon the foundation established by the first study, used the results 
of that study to develop a prototype user interface for the diagnostic system, and assessed 
that diagnostic system and user interface in helicopter operations. The goal was to 
determine the merits of aiding the aircrew with an automated diagnostic system when 
compared to the current, unaided situation. 

The approach was successful in identifying a number of important factors that can 
significantly influence the development of this kind of aiding technology, as well as raise 
general issues for consideration in the development of cockpit automation technology. 
First, the results clearly indicated that automated technologies can enhance aircrew 
performance when correctly designed, improving aircrew ability to diagnose true failure 
conditions and recognize false alarms. Second, the potential utility of certain types of 
information (and the type of aiding that information implies) was revealed, with pilots 
indicating primary interest (as measured by screen dwell time) in the ‘analysis’ category 
of information, as seen in Figure 2. In addition, the results showed differences in 
information requirements based on crew position. Third, aircrews were unanimous in 
their desire to have an electronic NATOPS as a means to access and to display 
emergency procedures. Fourth, the communication data indicated an insignificant effect 
of the automated diagnostic system on communication content or frequency, which 
implies that the technology does not significantly alter crew workload or crew 
coordination requirements as discussed in Bowers et al. (1995). 

1.2 Design of IE-NATOPS 

Design of IE-NATOPS focused on several issues: 

• aircrew access during an emergency with automatic triggering, 

• aircrew access during an emergency based on aircrew initiative and 
problem identification, 

• aircrew access during non-emergency periods, and 

• aircrew access to automatic and interactive performance data 
calculations. 

The first step taken to develop the design specification was to interview experienced 
Navy aircrews regarding the concept of an IE-NATOPS with specific reference to the 
Navy’s emerging CH-60S and SH-60R aircraft. Based on those interviews and an 
analysis of aircrew information needs, a design specification was formulated addressing 
both event-triggered needs and aircrew initiative information needs. Six modes of IE- 
NATOPS functionality were defined: checklists, NATOPS text and graphics, 
performance data, alerts, trends, and notes. Only the first four modes will be discussed in 
this paper since they are essential functions that are in current development, while 
assessments of the other functions (trends and notes) are being postponed pending design 
refinements. 
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Figure 2. Study 2 Results 

1,2.1 Initial Design 

The initial analysis focused on the SH60B NATOPS Flight Manual and associated pocket 
checklists and more recently on the SH-60F variant of the same information. The goal of 
the analysis was to review the overall structure of the NATOPS outline, identify the 
Parts, Chapters, Sections, and Sub-Sections that are used most by the aircrew in the 
cockpit, and to suggest potential design features for an effective interface structure. 

Where possible. Parts, Chapters, and Sections were combined under more general 
categories to streamline the overall NATOPS outline. This streamlining was necessary to 
minimize the number of potential selections on the interface. For example, categories that 
deal with aircrew requirements, training, and evaluation (i.e., Indoctrination, Aircrew 
Coordination, and NATOPS Evaluation) were combined under an “Aircrew 
Management” category. Care was taken to ensure the contents of the NATOPS were not 
changed. The outline structure of the NATOPS main body (i.e., Chapters, Sections, and 
Sub-Sections) remained unchanged and became the model for the breakdown structure of 
the information for the interface and the paths the aircrew can use when navigating 
through the information. Using MIL-STD-1472F, buttons, text fonts, color and the 
overall design of the interface screens were developed and presented in the form of 
storyboards. 

Included in the documentation analysis was a detailed analysis of the aircraft 
performance charts. This analysis was conducted to determine requirements for screen 
design and aircrew interaction. Each chart was reviewed to identify the particular 
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function it performed and to determine what information it required and what information 
it produced. 


The NATOPS analysis was conducted to develop an interface design that would not 
conflict with the current structure of the manual. The first goal of this analysis was to 
identify the information that would be used most frequently by the aircrews in the 
cockpit. After a review of the categories outlined in the manual, it was determined that 
Normal Procedures, Emergency Procedures, Performance Data, and All-Weather 
Procedures were required and that the remaining “Parts” of the document could be 
consolidated to limit the number of category selections incorporated in the interface. As 
shown in Figure 3, the required categories of information were maintained, and the 
categories that are common knowledge to aircrews (i.e., system descriptions, aircraft 
flight characteristics, training, aircrew requirements, etc.) were consolidated under the 
new categories shown in the two boxes on the left. 




Figure 3. Required Categories of Information 

The resulting six categories were incorporated into the interface design as the options on 
the NATOPS Main Menu (see Figure 4). 



Figure 4. IE-NATOPS Main Menu 
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A second goal of the NATOPS analysis was to develop an information structure for the 
interface to promote user acceptance. We concluded that the current outline structure of 
the NATOPS manual should be maintained and used as the information structure for the 
interface. It is important to maintain aircrew familiarity with the information in the 
document because any change to the current outline structure will make the information 
less familiar and greatly reduce aircrew acceptance of the system. Therefore, the Chapter, 
Section, and Sub-Section titles were used as the structure by which the aircrew navigated 
through the information. Knowing the basic information that each screen must display, it 
was then necessary to develop information selection paths that provide aircrews with the 
needed information with the least amount of button selection. Once the information 
selection paths were identified, screen designs were then developed. Figures 5 through 7 
provide examples of the results of this effort. Figure 5 is the screen that is displayed when 
the aircrew selects the Normal Procedures button on the NATOPS Main Menu. 



Figure 5. IE-NATOPS Normal Procedures Menu 

In the Normal Procedures menu (Figure 5.), the aircrew can select the desired NATOPS 
chapter that will display a menu (represented as a tree hierarchy as shown in Figure 6) 
listing all subsections of the chapter. 
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i Chapter 6 • Flight Preparation 
--] j 6.1 Mission Planning 

• 6.1.1 Weight And Balance 

* 6.1.2 Factors Affecting Range And Enduranc 

* 6.1.3 Planning For And Transport Of Cargo 
—1 _i 6.2 BriefingSDebriefing 

• 6.2.1 Briefing 

♦ 6.2.2 Oebrleflng Format 



Figure 6. Flight Preparation Menu Tree 

From the menu tree in Figure 6, the aircrew has the choice of selecting the desired 
NATOPS section, using the navigation button (i.e., Back, Forward, History, etc.), or 
choosing another mode. Figure 7 shows the screen that will be displayed when the 
aircrew selects a specific subsection. Alternatively, the user can follow a similar path 
through the hierarchy of checklist mode menus to access individual checklist procedures 
(Figure 8). 



1.1.1 Weight and Balance. 



Weight and balance control is necessary for 
correct center of gravity location and weight 
distribution. For safe and efficienl flight operations, 
the following weight and balance requirements are 
established: 


1. Proper loading shall be determined by 
use of the Weight and Balance Manual 
(NAVA1R 01-1B-40). 

2. Responsibility: Refer to NATOPS 
General Flight and Operaling Instructions 
(OPNAVINST 3710.7 series). 



6.1.1.1 Fuel Load 

Fuel load may be computed using the weight 
and balance manual. Minimum fuel reserves s 


Figure 7. NATOPS Text Screen 
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Figure 8. Checklist Text Screen 

As shown in Figures 4 through 8, the basic design concept provides a quick and efficient 
method of navigating through NATOPS and checklist information. When dealing with 
performance data, however, additional dynamic interactive capabilities are needed, thus, 
requiring a different design concept. Figure 9 illustrates the screen design that was 
developed for the “Ability to Maintain Level Flight, Single Engine” Chart. 
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Figure 9. Performance Data Calculations Screen 


The “Ability to Maintain Level Flight, Single Engine” performance chart provides the 
aircrew with the minimum and maximum velocities the aircraft should be flown to 
maintain level flight with one engine inoperable. These velocities are determined using 
the pressure altitude, outside ambient temperature, gross weight, and operating engine 
torque of the current aircraft flight profile. This information will be obtained from aircraft 
sensors, and the velocities can then be calculated automatically. Alternatively, the aircrew 
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can manually input any of the parameters to obtain the limiting velocities. For example, if 
the aircrew decides to fly at a higher pressure altitude, they would click on the PA field to 
clear and enter the new altitude via the keyboard. The velocities will be calculated 
automatically. The chart on the bottom of Figure 9 represents the Vmin and Vmax values 
over a range of gross weights. The arrows indicate the Vmin and Vmax for the gross 
weight shown on the screen. This chart can also be configured to show Vmin and Vmax 
across a range of pressure altitudes. The respective charts can be displayed by simply 
selecting the “Chart” button next to the PA or Gross weight text boxes. A set of tools are 
also being developed to aid pilots in their interactions with performance charts. These 
tools will be described in a later section of this report. 

1.2.2 IE-NATOPS Prototype 

Based on an iterative process that involved input from subject matter experts as well as 
end-users, the initial design was modified slightly. Perhaps the largest difference 
between the IE-NATOPS prototype and the initial design is in terms of how the main 
menu is set up for the NATOPS portion of IE-NATOPS. Specifically, in an effort to 
keep the prototype IE-NATOPS consistent with the look of the current NATOPS it was 
decided to keep the NATOPS “main menu” in the form of a table of contents (similar to 
how the hard copy NATOPS is currently designed). Thus, the NATOPS main menu 
changed from that illustrated in Figure 4 to that illustrated in Figure 10 below. 


n i - General Description 

H 7 - Systems 

□ 3 * Servicing and Handling 

S3 4 - Aircraft Operating Limitations 

83 6 - Aircrew Training, Qua!. & Requirements 


- Flight Pfparitlonl 


n 7 - Shore-aased Procedures 
n 8 - Shipboard Proc edures 

□ 9 - Special Procedures 
B 10 * Functional C fiocMllgnt 

n 11* Plight Techniques and Characteristics 
S3 12 - Emergency Procedures 

□ 13 - instrument Procedures 

n 14- Extreme Weather Operations 
63 15-Avionics 

63 id - Communication Equip, and Procedures 

63 17 - Navigation 

63 10 - Armament Systems 

63 19 - Coordination Procedures 


M 


Figure 10. Prototype NATOPS main menu 

Another benefit to the current configuration of the main menu is that it reduces the 
amount of button pushing required in order to get to needed information. In the initial 
design, information of this nature would have been found two levels down. Information 
here is organized in a tree fashion in which participants wishing to obtain more 
information simply expand the major chapter headings by clicking on the “+” symbol on 
the left. For example, expanding the “Systems” chapter reveals the subordinate subtopics 
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(see Figure 11). Those sub-topics with to the left of the topic indicate that they are 
fully expanded, while those with a “+” can be further expanded. 



H 2.1 SYSTEMS 

2.1.1 Engine. 

2.1.2 Engine Control System. 

2.1.3 Engine Fuel System. 

2.1.4 Engine Electrical System. 

2.1.6 Engine Operation Summary. 

2.1.6 Engine Oil System. 

2.1.7 Engine Start System. 

2.1.8 Engine and Inlet Anti-ice System. 

2.1.9 Engine Parameter Sensing. 

2.1.10 Engine Instruments. 

C3 2.2 AUXILIARY POWER UNIT SYSTEM 
H 2.3 FUEL SYSTEM 
Q 2/4 TRANSMISSION SYSTEM 
B 2.5 ROTOR SYSTEMS 

2.5.1 Main Rotor System. 

2.5.2 Tall Rotor System. 

2.5.3 Rotor Brake System. 

Q 2.6 ELECTRICAL SYSTEM 


Figure 11. Prototype Systems Menu Tree. 


Once the user fully expands a specific subsection, information is illustrated as it was with 
the initial design (see Figure 12 and Figure 7). Furthermore, as with the initial design, 
the user can also follow a similar path through the checklist mode hierarchy menus to 
access individual checklist procedures (see Figure 8). Performance data calculation 
screens are also similar to that of the initial design (see Figure 9) within this later 
prototype. 
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Figure 12. Prototype NATOPS Text Screen 

1.3 IE-NATOPS Benchmark Evaluation 

A benchmark evaluation of the IE-NATOPS design was conducted in the H-60 rotorcraft. 
As part of that evaluation, a detailed knowledge engineering effort was conducted in the 
last quarter of 2000 with pilots at VX-1 (NAWCAD) to identify current usage of the 
NATOPS and Pocket Checklist (PCL) while airborne (Deaton, Burke, & Good, 2000). 
Questionnaire and interview protocols were developed and data were obtained to identify 
and prioritize specific tasks accomplished with the use of the NATOPS and PCL. This 
effort also addressed information requirements associated specifically with graphics 
usage. Results from the knowledge engineering study were used to define the criterion 
tasks for the design of the benchmark evaluation discussed in this report. Major 
participating organizations in this effort have included CHI Systems, Inc., Honeywell 
Laboratories, Pennsylvania State University Applied Research Laboratory, the University 
of Central Florida, the Naval Air Warfare Center Training Systems Division 
(NAWCTSD), and the Office of Naval Research (ONR). 

The purpose of the benchmark evaluation was to compare aircrew performance when 
using the traditional paper copy of NATOPS and the Pocket Checklist (PCL) with that of 
the prototype IE-NATOPS. Prior research addressing the integration and interpretation 
of text information presented either on paper or in electronic form has found significant 
differences favoring paper-based presentation of text in terms of search times and 
comprehension (Rice, 1994; Gould et ah, 1987; Askwall, 1985). Though search times 
seemed to be shorter with paper-based text presentation compared to electronic 
presentation, readers have been found to search almost twice as much information when 
using paper-based presentation of text than when using electronic presentation (Askwall, 
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1985). Possibly related to these findings, Rice (1994) has found that reading 
comprehension is maximized when paper-based presentation of text is used compared to 
electronic presentation. 

Both the Askwall and Rice studies used short stories as the source of their stimuli in 
laboratory settings, with university students as participants. When an electronic checklist 
was used as the source of information in training flight simulations with aviators, fewer 
subsystem failures were detected by the aviators than when a paper checklist was used 
(Palmer & Degani, 1991). When the checklist was automated, with the automated 
checklist checking aircraft systems and automatically highlighting missed steps, the 
aviators' detection of subsystem failures was even less likely than for the manually sensed 
electronic checklist. Related to these findings, Mosier et al. (1992) found that trouble¬ 
shooting was less accurate when electronic formats of the checklist were employed than 
when the paper format was employed. In addition, aircrews communicated less about the 
status of the aircraft when using the electronic checklist formats than the paper formats. 
Finally, recent work conducted by Boorman (2000) has shown that the B777 electronic 
checklist design developed by Boeing has reduced flight crew errors. However, the 
author pointed out that it could have produced new error modes were it not for explicit 
efforts to avoid such problems. 

The results of studies performed to date reveal mixed results when using electronic 
formats for the aircraft checklist. In an attempt to clarify this issue and to establish the 
problem-solving performance of aviators using IE-NATOPS, a benchmark evaluation 
was performed. As with past studies, this benchmark evaluation attempted to establish 
whether or not there are notable differences in resolving problem situations dependent 
upon the format (electronic vs. paper) of the NATOPS and PCL used. Within the context 
of the benchmark evaluation two studies were conducted: (a) one involving 
experimenter-directed searches for information and so providing a measure of pure 
physical access time to navigate through the relevant interface to reach the desired 
informations, and (b) one involving self-directed searches for information needed to 
resolve a problem as a measure of search times that incorporate the variations due to 
search strategies. The results of this evaluation will be used to guide further development 
of IE-NATOPS. 

1.4 NATOPS Graphic Evaluation 

As paper manuals are converted into electronic display systems, the issue of how to 
display large figures and schematics on small displays is of considerable concern. 

Modern advances in the design of information systems have resulted in more capable, 
lightweight, low-power, high-resolution, portable information devices. The availability of 
powerful and cost-effective technology has fueled the trend towards putting more and 
more information into electronic form. The trend of converting paper materials into 
electronic form, however, may not always be a complete solution (Frey, Rouse, & Garris, 
1992). A piece of paper has hundreds of times the resolution of an electronic display, so 
figures and pictures often cannot be shown at their full resolution all at once. The trend 
towards smaller display screens on portable devices only exacerbates the problems of 
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resolution and available screen real estate on which to display graphic figures and 
schematics. When this situation arises, the question then becomes 1) how best to initially 
display the image, and 2) how to allow the user to manipulate and interact with the 
image. Some of the logical solutions for interacting with an image might be to provide 
the functionality of zooming, panning, or scrolling which would provide the user with a 
small “window” through which to view the image. Scrolling and panning allow the user 
to maintain the field of view, but changes the portion (or “window into”) of the figure 
being viewed. Zooming decreases the field of view, but maintains the portion of the 
display that is being viewed. In addition, methods of branching (i.e., hyperlinks) allow 
users to change the field of view, but can involve changing the representation of the 
entire system. Each of these techniques would provide additional representations to the 
user that might allow them to interact with the image more effectively. However, these 
solutions have limitations. For example, changing locations in an image with any of 
these techniques might become tedious or, more importantly, might cause the user to lose 
his/her focus of the “big picture” that the image represents. 

The problem of usability of graphics information in IE-NATOPS has been of particular 
concern throughout the development of this concept. As the NATOPS manual content is 
digitized and moved onto electronic displays, the question arises of how to most 
effectively view large pictures. Electronic screen real estate is limited, so it becomes 
impractical to display large pictures on small screens. One possibility for continuing to 
provide all the information in the original graphic is simply to scale the image such that it 
could be accommodated by the display device. However, simply scaling down the image 
can potentially render it unreadable. This problem will likely be exacerbated in the 
context of the electronic kneeboard that is currently planned as the IE-NATOPS’ host, 
since this device offers screen size options only in the range of 6-inch to 8.5-inch 
(diagonal). Furthermore, cockpit vibration may significantly exacerbate this situation. 
Accordingly, we have developed a variety of concepts for an EE-NATOPS Graphics 
Interaction Tool (GIT) that will enable the user to have access to graphic information 
needs in a way that permits adaptive software to reconfigure the relevant graphic 
information and achieve effective presentation within the display constraints. 

In addition to the benchmark evaluation description, this report will identify the problems 
associated with displaying and interacting with graphical images that are presented in 
paper NATOPS manuals. Also provided is a description of a set of graphical interaction 
tools that have been conceptualized to enhance the usefulness of information provided to 
the pilots in various types of charts found within NATOPS 

1.5 Organization of Report 

The body of this report presents the methods used in the benchmark evaluation and the 
results and discussion of all facets of the study. Appendices in the back of this report 
contain the paperwork used to brief, debrief, and acquire biographical information about 
the participants. Appendices also include additional details on several of the analyses 
reported in the main body of the report. The IE-NATOPS training protocol is also 
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provided in the appendices, as are the evaluation scenarios, a preliminary task analysis of 
the scenarios, subject notes, and the post-evaluation questionnaire. 
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2. Methods 


2.1 Apparatus 

This benchmark evaluation employed a paper-based simulation of the information search 
and decision processes involved in performing missions requiring the use of the H-60F 
helicopter’s NATOPS information in the context of scripted flight scenarios. Briefing, 
debriefing, biographical, and post-experiment questionnaires, training materials, and 
experimental scenarios were all paper-based. Both a paper version of the NATOPS 
Pocket Checklist (PCL) and an electronic version of the NATOPS represented by the EE- 
NATOPS/PCL were employed. 

The IE-NATOPS system and interface were hosted on a PC computer and standard 
monitor. Although this study employed a desktop PC, eventually we envision that 
processing and display will be via a dedicated “electronic kneeboard”, which consists of a 
pen-based, flat panel display that mounts on the pilot’s thigh and is connected by wire to 
a chassis mounted processor with a removable storage device. The IE-NATOPS 
interface was scaled down to the 6in. display size of the future kneeboard device. A 
mouse was used to make all interactions with IE-NATOPS. It is actually expected that 
airborne use of IE-NATOPS would more likely be via a touch screen using a stylus, and 
the interface has been designed for touch-stylus operation; however, performance 
differences between mouse and stylus operation are expected to be negligible in the 
context of this study because relatively few pointing actions are required and those 
actions will consume very little time relative to the composite performance times to be 
analyzed. 

Participant biographical data (see Appendix D), and post-experiment questionnaire forms 
(see Appendix E) were developed and presented to the participants in this study. The 
participant biographical data form gathered information on the participants’ flight 
experience. The post-experiment questionnaire queried the participant’s opinions of the 
experiment, training, and NATOPS displays. 

Training materials were developed and can be examined in Appendix G. The procedure 
section (Section 2.5) will describe the specific training protocol used to train participants 
on the IE-NATOPS display. 

2.2 Participants 

Ten fleet replacement pilots stationed at Naval Air Station Jacksonville participated for a 
period of 2.5 to 3 hours each. All Naval aviators participating in this study had current 
experience in the H-60F. Table 1 summarizes pilot background/flight experience as well 
as numerous other demographic variables. 
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Table 1. Biographical Information and Related Flight Experience 


Demographic Information/ 

Flight Experience 

Response 

Rank 

LT=7 LCDR=3 

Sex 

Male=10 

Age 

M=31.8 yrs 

Time in Service 

M=125.8 mos. 

H-60F Flight Time 

M=855.0 hrs 

H-60F Simulator Flight Time 

M=29.5 hrs 

Flight Time (last 90 days) 

M=188.8 hrs 

Flight Time Wing 

M=1782.0 hrs 

Total Flight Time 

M= 1909.0 hrs 


2.3 Experimental Design 

The main question to be answered by this benchmark evaluation was whether there are 
notable differences in performance dependent on the search tool used (i.e., paper or 
electronic NATOPS). As access to aviators was limited, this study employed a within- 
subjects design to examine the effects using the paper-based version of the 
NATOPS/PCL as compared to using IE-NATOPS. In an attempt to control for possible 
order effects, the method of presenting the NATOPS and PCL information (i.e., paper or 
electronic) was counterbalanced. 

Within the overarching framework of this benchmark evaluation, participants completed 
two experiments that built upon one another. The first experiment served two purposes: 
(a) train participants on the use of IE-NATOPS and (b) determine whether access time 
differed using IE-NATOPS as compared to the traditional paper NATOPS and PCL. The 
dependent variable, access time, was defined as the length of time it took participants to 
find particular information when guided by the experimenter. The second experiment 
examined a different aspect of participant performance by looking at the length of time it 
took participants to locate what they felt were the answers to specific problem scenarios 
that were presented. Thus, the primary dependent variable in the second experiment was 
defined as the total time it took participants to self-direct themselves in the search of what 
they felt was the path to the correct answer for each of the problems presented. 
Performance in the second experiment involved both the time it took participants to 
search for the information, as well as the time it took to actually process the information 
into answers to the scenario problems. Both experiments compared performance using 
the paper NATOPS/PCL to performance with the electronic version of the 
NATOPS/PCL (IE-NATOPS). 

The other-directed trials (Experiment 1) and the self-directed trials (Experiment 2) 
required participants to do slightly different tasks. Within the other-directed trials, we 
were interested solely in examining the time it took for participants to access the 
information using alternative versions of NATOPS (paper and electronic). Accordingly, 
participants were told step-by-step where to go. This required only that they remember 
how to use the decision-aids to which they were currently exposed (paper-based or IE- 
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NATOPS). It was felt that this would be akin to a situation where participants knew 
exactly where they needed to go to find the information, reflecting pure access time. This 
also provided a further assessment of the effectiveness of training. Conversely, the self- 
directed trials reflected a combination of search strategies and access rates. Participants 
had to decide how to navigate through the interface (paper or electronic) to find the 
desired information (search strategy), as well as to execute that strategy to arrive at the 
appropriate end-point. Thus, the two experiments, though connected, reflected different 
types of aviator performance. Furthermore, all participants completed the other-directed 
experiment assessing pure access rates prior to the self-directed experiment that also 
involved navigation strategy, as the ability to access the information was fundamental to 
the more complex performance reflected in the second experiment. 

2.4 Procedure 


Participants were given: (a) an experiment briefing (Appendix A), (b) an informed 
consent form (Appendix C), (c) a privacy act statement (Appendix B), and (d) an 
anonymous demographic questionnaire (Appendix D) that covered both experiments. 
Upon completion of the briefing and demographic questionnaire, participants began the 
experimental session(s). 

2.4.1 Experiment One: Other-Directed Search 

The order in which participants received the two formats of the NATOPS and PCL (paper 
version or IE-NATOPS) was counterbalanced. The procedure that is described below is 
for participants who received the paper version of the NATOPS/PCL first, followed by 
the IE-NATOPS condition. For participants who received the IE-NATOPS condition 
first, followed by the paper condition, the order of the procedure was reversed. 
Specifically, participants were trained on the use of IE-NATOPS, completed the searches 
using this tool, and then completed searches using the paper version. See Figure 13 for a 
depiction of the flow of events, dependent upon the order of counterbalancing. 


Paper NATOPS First, then IE-NATOPS 


Paper NATOPS 

IE-NATOPS 

IE-NATOPS 

Evaluation 

Training 

Evaluation 


IE-NATOPS First, then Paper NATOPS 


IE-NATOPS 

IE-NATOPS 

Paper NATOPS 

Training 

Evaluation 

Evaluation 


Figure 13. Experiment 1 Timeline 
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For individuals assigned to the paper-based condition, the experimental procedure was as 
follows. Participants were instructed by the experimenter to go to specific sections of the 
NATOPS/PCL, verbalizing their path along the way. The searches that participants 
completed required them to transverse through multiple sections of the paper-based 
NATOPS/PCL (see Appendix H). Two evaluators were seated in close proximity to 
record when participants reached their final destination, as indicated by the experimenter. 
No training was given on the use of the paper-based NATOPS /PCL for it was assumed 
that pilots were well trained on the use of this format as a result of regular NATOPS 
exams and daily use. Following performance on the paper-based version, participants 
were asked if they had any remaining questions on what they had just completed. 

Participants were then told that the next section of the study would require them to use an 
electronic version of the NATOPS/PCL and that, prior to using the electronic version, a 
short training session would be conducted to enable them to operate the electronic 
version. Training on the use of IE-NATOPS was an interactive hands-on process that 
was led by the experimenter. Training proceeded in a manner such that participants 
learned general information about the design and structure of IE-NATOPS followed by 
instruction on each of the specific modes (i.e., checklist, data, alert, and NATOPS) 
contained in IE-NATOPS. Participants were then given ten minutes to practice a list of 
objectives that required them to transverse among the different modes of IE-NATOPS. 

At the completion of the ten minutes, participants were asked if they had any questions or 
concerns. Finally, the experimenter asked participants to traverse through the database to 
find several specific items of information. Participants were presented with progressively 
more complex tasks using IE-NATOPS, each of the tasks building upon previous tasks. 

In this way, participants received practice in the basic navigation skills required to access 
information in IE-NATOPS. This last practice session was akin to what participants 
would be required to do on subsequent experimental trials. See Appendix H for a copy of 
the script used during this training. 

Total training time on IE-NATOPS lasted approximately thirty minutes for each 
participant. This duration of training was determined to be adequate through pilot trials. 
Participants were allowed to progress forward if they felt comfortable with the interface 
and the experimenter judged that they were adequately manipulating the interface so that 
the actual manipulation of the interface would not be a problem during the experiment. 

At the completion of training, participants were told that they would begin the portion of 
the study that required them to locate specific information, as dictated by the 
experimenter, using IE-NATOPS. Participants were to follow the same procedures that 
were followed for the searches using the paper-based version of the NATOPS/PCL. See 
Appendix H for further detail on the specific searches required. Aaccess time was 
computed by two evaluators using stopwatches. At the conclusion of the first 
experiment, participants were informed that they could take a short ten-minute break and 
upon their return they would begin the second experiment (as described in the pre-brief 
and informed consent). 
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2.4.2 Experiment Two: Self-Directed Search 

As with the first experiment, the format of the NATOPS and PCL was counterbalanced. 

In addition, problem scenarios were also counterbalanced. Upon return from break from 
their sessions for the Experiment One, each participant was again seated at a desk and 
presented with written problem scenarios, one at a time, totaling ten scenarios. Each 
participant was asked to resolve each of the ten problem scenarios using the NATOPS 
information for the H-60F (Appendix I). For five of the scenarios the participant was 
restricted to using the paper NATOPS/PCL, while the remaining five scenarios involved 
access to NATOPS/PCL information through the use of IE-NATOPS. The procedures 
used by the participants to resolve the problem scenarios were directed to be those 
currently used by these naval aviators in their aircraft and are explicitly stated in the H- 
60F NATOPS. See Appendix I for further detail. 

Experimenters observed how the participants resolved the specific problems in each 
scenario, noting how long it took participants to reach a solution and the outcome of that 
solution (i.e., accuracy). Once participants had read a scenario, they were given the 
opportunity to ask any questions relative to the problem being presented. Participants 
were asked to state verbally how they would resolve the problem situation (i.e., what 
information they would access within NATOPS/PCL). Once that process was completed, 
participants were instructed to trace the path within the NATOPS/PCL that would lead 
them to their correct answer. Timing (via hand-held stopwatches) began once each 
individual verbally stated their plan for accessing the needed information and indicated 
that they were starting the active navigation process. The completion of the response 
time was defined as the time when the participant verbally stated the solution by 
accessing it on the screen or in the paper version of the NATOPS/PCL. 

2.4.3 Debriefing 

After participants completed both experiments, as well as the post-experiment 
questionnaire, they were debriefed as to the overall purpose of the studies (see Appendix 
F). 

2.4.4 Graphics Evaluation 

Following the debriefing, the evaluation of graphical concepts and tools proposed for use 
within IE-NATOPS took place. Each participant was told that they would now be shown 
a variety of concepts that included tools for interacting with large graphical images 
within NATOPS. This portion of the evaluation was set up in an interview format. The 
experimenter described each concept and then solicited feedback from each participant 
on each of the concepts. Upon completion of this portion, participants were thanked for 
their participation and any questions that remained were answered. 
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3. Results 


3.1 Descriptive Information 

A scan for outliers (scores more than +/- 3 standard deviations from the mean) resulted in 
three individuals being deleted in analyses of experimenter-directed search data and two 
individuals being deleted in analyses of self-directed search data. 

Correlations between key study variables and demographic variables are presented in 
Appendix J (Table J-2). Overall, demographic variables were not correlated with key 
study variables, but there were two exceptions. Within Experiment 1, amount of time 
spent in the 60H in the last 30 days was positively correlated with access time using the 
paper copy of the NATOPS/PCL. Within Experiment 2, amount of time spent in the H- 
60F simulator was positively correlated with search time using the electronic version of 
the NATOPS/PCL (IE-NATOPS). Thus, it is possible that the amount of time spent in 
the 60H in the last 30 days and the amount of time spent in the 60F simulator might 
explain a portion of the variance accounted for in other-directed and self-directed search 
times, respectively. 

3.2 Experiment 1: Other-Directed Search 

To investigate the impact of search mode/format on access time, a 2 (search mode) x 6 
(trials) repeated measures ANOVA was conducted. Within this analysis, amount of time 
spent in the H-60H in the last 30 days was used as a covariate. With the covariate 
entered in, results indicated a main effect for search mode (F(l,5)=97.05, pc.Ol, 
eta 2 =.951). Specifically, when participants used the electronic search mode/IE-NATOPS 
(M=56.55, SE=2.01), access times were significantly quicker than when the paper search 
mode (hard copy NATOPS/PCL) was used (M= 106.26, SE=4.07). See Figure 14. 
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Figure 14. Mean Access Time by Search Mode 
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Results also indicated a main effect for content-access trial (F( 1.481, 7.404)= 16.80, 
pc.Ol, eta 2 =.771), indicating that participants had significantly different access times 
dependent on the particular content-access trial (the Mauchly’s Test of Sphericity was 
significant, thus the Greenhouse-Geisser correction was used). See Figure 15 for a 
pictorial view of this result. 



Figure 15. Mean Access Time by Content-Access Trial 

The intention of this portion of the study was to design content-access trials to cover a 
range of actions and to ensure that participants were capable to operate the electronic 
search tool (IE-NATOPS). It was not to examine individual trials. Hence, a priori 
hypotheses were not formulated regarding the effect of individual trials. However, given 
the significant main effect, post-hoc comparisons were conducted using a modified 
Bonferroni correction (as suggested by Kinnear & Gray, 1997). Due to the corrections 
made, a more conservative p value must be used (as compared to the normal p of .05 or 
.01). Specifically, in order for a comparison to be deemed significantly different, the p 
value must be less than .003. Given this, the following trials were found to represent 
significant differences: (1) Trial 2 vs. Trial 3, (2) Trial 2 vs. Trial 5, and (3) Trial 2 vs. 
Trial 6. The differences between access times during Trial 2, as compared with most 
other trials, makes intuitive sense in that the information that participants were directed to 
access in Trial 2 represented a short two-step procedure, whereas the other trials were 
more complex (i.e., required more steps). Although the graphical depiction above is 
deceiving, in that it looks like the difference between Trial 2 and Trial 1 would also be 
significant, access times during Trial 1 experienced a great deal of variation. 

Specifically, the access times in Trial 1 had a standard error of 15.60, and a 95% 
confidence interval of 76.98 to 157.17. The reported standard error of all other trials 
ranged from a minimum of 2.6 (Trial 2) to a maximum of 8.78 (Trial 6). 
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Finally, a non-significant interaction (F(2.119,10.597)=2.88, p>.05, eta 2 =.365) was 
found indicating that search mode and content-access trials do not interact to produce 
different participant access times (Greenhouse-Geisser correction used). 

3.3 Experiment 2: Self-Directed Search 

Experiment 2 was conducted to examine whether search mode (paper or electronic) had 
an effect on participant self-directed search time. The design of the second experiment 
was set up slightly differently than the first experiment, precluding a factorial ANOVA, 
as used in experiment 1. 


3.3.1 Primary Analyses 

Results of a repeated measures ANOVA, using time spent in the H-60F simulator as a 
covariate, indicated that search mode did not significantly impact search time 
(F(l,)=3.46, p>.05, eta 2 =.409). Indicating that, on average, search mode did not affect 
the time it took for participants to search for the required information and then once 
found, process that information (paper M= 184.23, SE=20.06; electronic M= 164.51; 
SE=18.16). While not significant, results did show a more rapid response for participants 
using the electronic mode. 


Because participant’s searches were self-directed, a second set of analyses were 
conducted to examine whether search mode had an effect on the accuracy of participant 
responses to each problem scenario, as judged by a subject matter expert (SME). 
Examination of summary statistics clearly shows that there was very little variance in 
accuracy rates. In fact, for both search modes, participants tended to answer correctly 
94% of the time. See Table 2 for more detail. Appendix J provides additional detail on 
specific subject accuracy performance by scenario. 

Table 2. Accuracy of Response (Summary Statistics) 



Mean 

Standard 

Deviation 

Minimum 

Maximum 

N 

Scenario 1 

Paper 

1.00 

.000 

1.00 

1.00 

4 

Electronic 

.75 

.500 

0.00 

1.00 

4 

Scenario 2 

Paper 

1.00 

.000 

1.00 

1.00 

2 

Electronic 

1.00 

.000 

1.00 

1.00 

6 

Scenario 3 

Paper 

1.00 

.000 

1.00 

1.00 

4 

Electronic 

1.00 

.000 

1.00 

1.00 

4 

Scenario 4 

Paper 

1.00 

.000 

1.00 

1.00 

2 

Electronic 

.83 

.408 

0.00 

1.00 

6 

Scenario 5 

Paper 

1.00 

.000 

1.00 

1.00 

4 

Electronic 

1.00 

.000 

1.00 

1.00 

4 
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Scenario 6 


Paper 

1.00 

.000 

1.00 

1.00 

4 

Electronic 

1.00 

.000 

1.00 

1.00 

4 

Scenario 7 

Paper 

.83 

.408 

0.00 

1.00 

6 

Electronic 

1.00 

.000 

1.00 

1.00 

2 

Scenario 8 

Paper 

1.00 

.000 

1.00 

1.00 

4 

Electronic 

1.00 

.000 

1.00 

1.00 

4 

Scenario 9 

Paper 

.67 

.516 

.000 

1.00 

6 

Electronic 

.75 

.500 

.000 

1.00 

2 

Scenario 10 

Paper 

1.00 

.000 

1.00 

1.00 

4 

Electronic 

1.00 

.000 

1.00 

1.00 

4 


3.3.2 Ancillary Analyses 

Within the second experiment, each participant completed 10 scenarios that were 
counterbalanced across search mode. For each participant, five of these scenarios were 
completed via a paper search and five via an electronic search. The study was designed 
in this manner due to the fact that the intent was not to examine individual scenarios, but 
how search mode affected search times across scenarios (accordingly, SMEs had 
designed and assessed the scenarios to exhibit similar complexity). However, 
observations during the study suggested that there may have been a slight interaction 
between search mode and individual scenarios. Because of design constraints, the 
authors felt that statistical analyses were not appropriate in this instance. However, an 
examination of descriptive information, broken down by search mode, may provide 
insight into trends and areas of future research. Table 3 presents these data in tabular 
form and it is illustrated graphically in Figure 16. The figure presents these data in terms 
of error bars extending one standard error above and below the mean in each case. 

Table 3. Search Time for Problem Solving Trials: Trial x Search Mode 



Paper 

Electronic 

Scenario 1 

M=178;SE=27 

M=77;SE=23 

Scenario2 

M=170;SE=217 

M=84;SE=42 

Scenario3 

M=131;SE=14 

M=34;SE=12 

Scenario4 

M=200;SE=113 

M=265;SE=70 

Scenario5 

M=274;SE=92 

M=214; SE=108 

Scenario6 

M=103;SE=26 

M=104; SE=30 

Scenario? 

M=224;SE=47 

M=458; SE=242 

Scenario8 

M=187;SE=30 

M=219; SE=36 

Scenario9 

M=174;SE=40 

M=196;SE=64 

Scenario 10 

M=202;SE=86 

M=221;SE=73 
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Mode PE PE PE PE PE PE PE PE PE PE 
Scenario #1 2 3 45 6 78 910 

Figure 16. Search Time for Problem Solving Trials: Error Bars 

In examining the descriptive information presented above, perhaps the first thing that the 
reader needs to remember is that, as the above means and standard errors represent 
participant search times, smaller numbers indicate more rapid response times. In 
addition, there are a few trends that might provide some additional insight into current 
findings. In terms of the scenarios themselves, the data indicate that some scenarios 
witnessed extremely large standard deviations (see scenarios 2,4, 5, and 7). The large 
variability in participant responses within these scenarios makes it difficult to determine 
the effect of search mode. Secondly, although scenarios were not designed to target 
specific characteristics, but to cover a range of problems that pilots might face, a cursory 
analysis can be made as to how scenario content may contribute to the above patterns. At 
a general level, the following observations can be made. The scenarios in which 
participants had quicker search times via the electronic search mode generally had the 
following characteristics: (1) required use of a performance chart or (2) involved a 
checklist that required participants to go to another section/checklist (hyperlink 
provided). Moreover, the scenarios in which participants’ search times were quicker 
using the paper search mode tended to be searches that involved: (1) finding tables or (2) 
general text. Appendix L provides a preliminary task analysis of the 10 scenarios, and 
may shed some light on how scenario characteristics interact with presentation mode. 
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3.4 Post-Experiment Questionnaire 


Table 4 presents the mean response rates for the post-experiment questionnaire (see 
Appendix E for detailed wording of each question). Questions 1-4 dealt primarily with 
the IE-NATOPS training that was afforded each participant during the first experiment. 
Questions 5 and 11-12 were associated specifically with aspects of the IE-NATOPS 
interface. Questions 6-10 compared paper NATOPS and IE-NATOPS on numerous 
dimensions. Note that different scales and scale ranges were used for these questions, in 
particular with Questions 1-5 using three-point scales and Questions 6-12 using five- 
point scales; all reported comparisons are within a single scale type. 

Responses on Questions 1-4 indicated that participants, in general, agreed that IE- 
NATOPS training was well organized, thorough, well-paced, and included sufficiently 
complex practice problems to afford participants an adequate training opportunity. 
Responses on Question 5 showed that participants felt confident regarding their use of 
IE-NATOPS for acquiring information in flight. Questions 11-12 provided a measure of 
ease of use of numerous functions of the IE-NATOPS interface. In general, all 
participants agreed (or strongly agreed, in some cases) that all specified interface 
functions were easy to use. Those functions that were rated as the easiest to use were the 
tab (i.e., “file tabs” allowing selection from multiple simultaneously active checklists) 
feature within the checklist section and the performance charts. Those functions that 
were not quite as easy to use included the history function and the bookmark function. 

Questions 6-10 consisted of a comparison between NATOPS and IE-NATOPS, and, thus, 
provided the opportunity to conduct a more quantitative/statistical comparison between 
participants’ responses. Accordingly, five related-measures t-tests were conducted on 
these five matched questions. Results of this analysis indicated that the only comparison 
that was significant was that between 6a and 6b. In this case, participants were 
significantly more likely to agree with the statement that the IE-NATOPS interface, in 
comparison to the traditional paper NATOPS, was easier to manipulate, [t(9) = 2.25, p = 
.05]. The only other comparison that approached significance was that between 10a and 
10b, [t(9) = 2.09, p = .06]. That is, participants rated IE-NATOPS as easier when 
accessing information involving searching in multiple sections. While this difference 
was not statistically significant (M = 2.2 for NATOPS vs. M = 1.5 for IE-NATOPS), it 
was in the anticipated direction. Given the nature of electronic media and their efficient 
searching capabilities, the research team had expected IE-NATOPS to facilitate access to 
information contained in physically separated sections of the traditional NATOPS. All 
other comparisons between NATOPS and IE-NATOPS were insignificant. However, 
with the exception of one question, all survey questions favored IE-NATOPS in 
comparison to NATOPS in terms of ease of use. The one exception was between 8a and 
8b. Although not statistically significant, the means indicated that participants were less 
likely to agree with the statement that they had no trouble reading and understanding the 
NATOPS data when using the IE-NATOPS interface. 
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Additional analyses were performed to address potential differences on the survey 
questions as a function of specific grouping variables obtained from the demographic 
data. Accordingly, data from the survey questions were re-analyzed using the following 
category variables: 

1) rank—LT vs. LCDR, 

2) age—less than or equal to 30 years, vs. greater than 30 years., 

3) time in service—less than or equal to 105 months vs greater than 105 
months, 

4) total hours in the H-60F—less than or equal to 600 hours vs. greater than 
600 hours, 

5) approximate time in the H-60F in the last 90 days—less than or equal to 
15 hours vs. greater than 15 hours, 

6) approximate time in the H-60F simulator—less than or equal to 100 hours 
vs. greater than 100 hours, 

7) total time in rotary wing—less than or equal to 1500 hours vs. greater than 
1500 hours, and 

8) total flight time—less than or equal to 1600 hours vs. greater than 1600 
hours. 

Note that with the exception of rank (there was little variability here — participants were 
either LT or LCDR), the criteria for determining the two groupings for each category was 
based on median values for each category. That is, in most cases, 50% of the participants 
were in each grouping (e.g., for the age category, half of the participants were less than 
30 years, while the other half were greater than 30 years). Traditional t-tests for 
independent groups were conducted on each of these categories for the particular groups 
that were developed. Detailed results for these ancillary analyses are presented in 
Appendix K. 

The additional analyses involved comparisons for all 23 of the items and sub-items of the 
questionnaire and across all eight grouping variables enumerated above. Thus, a total of 
184 individual comparisons were made at this stage. However, since the eight groupings 
are highly redundant and overlapping with one another (i.e., individuals with higher rank 
tend to be older, with more time in service, more flight hours, etc.), these do not represent 
independent comparisons. Out of all of these comparisons, only three were found to be 
statistically significant. These are: 

• Item 8a [t(8) = 2.45, p = .04] indicates that the pilots with more H-60F 
flight time agreed more strongly with the statement that they had no 
trouble in reading and understanding the NATOPS data using traditional 
NATOPS as a source. 

• Item 1 lb [t(8) = 2.89, p = .02] indicates that pilots with more experience 
were not as likely to agree that the bookmark function was easy to use. 

• Item 12a [t(8) = 2.97, p = .02] supports 1 lb in indicating that more 
experienced H-60F pilots did not find the bookmark function as useful as 
less experienced H-60F pilots in the operational environment. 

Additionally, non-significant trends for the grouping factors of rank, age, and time in 
service suggested that more senior pilots (in rank, age, or time in service) rate IE- 
NATOPS higher for ease of interface manipulation than the more junior officers. 
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However, also non-significantly, more junior pilots tended to report less trouble reading 
and understanding IE-NATOPS. 


Table 4. Overall Mean Response Rates for Post-Experiment Questionnaire 


Item 

Scoring Scale 

Mean 

SD 

1. IE-NATOPS training organization 

1 = too loose 

2 = perfect 

3 = too regimented 

2.10 

.32 

2. Amount of 

IE-NATOPS training 

1 = too little 

2 = sufficient 

3 = too much 

2.10 

.32 

3. Amount of 

IE-NATOPS practice 

1 = too little 

2 = sufficient 

3 = too much 

2.10 

.32 

4. Difficulty of IE-NATOPS training 
scenarios 

1 = too easy 

2 = sufficient 

3 = too complex 

2.10 

.32 

5. Confidence for in-flight use of IE- 
NATOPS 

1 = very confident 

5 = very uncertain 

1.60 

.52 

6a. Paper NATOPS interface easy to 
manipulate 

1 = strongly agree 

5 = strongly disagree 

2.80 

1.48 

6b. IE-NATOPS interface easy to 
manipulate 

1 = strongly agree 

5 = strongly disagree 

1.50 

.53 

7a. Always knew how to use paper 
NATOPS interface 

1 = strongly agree 

5 = strongly disagree 

2.20 

1.32 

7b. Always knew how to use IE-NATOPS 
interface 

1 = strongly agree 

5 = strongly disagree 

1.90 

.57 

8a. Had no trouble reading & 
understanding paper NATOPS 

1 = strongly agree 

5 = strongly disagree 

1.30 

.48 

8b. Had no trouble reading & 
understanding IE-NATOPS 

1 = strongly agree 

5 = strongly disagree 

1.60 

.70 

9a. Had no trouble finding data with paper 
NATOPS 

1 = strongly agree 

5 = strongly disagree 

2.50 

1.35 

9b. Had no trouble finding data with paper 
NATOPS 

1 = strongly agree 

5 = strongly disagree 

2.10 

.32 

10a. Easy to search multiple sections with 
paper NATOPS 

1 = strongly agree 

5 = strongly disagree 

2.20 

1.03 

10b. Easy to search multiple sections with 
IE-NATOPS 

1 = strongly agree 

5 = strongly disagree 

1.50 

.53 

11. For IE-NATOPS: 




11a. History function easy to use 

1 = strongly agree 

5 = strongly disagree 

2.20 

.79 

1 lb. Bookmark function easy to use 

1 = strongly agree 

5 = strongly disagree 

2.10 

.74 
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1 lc. Navigation Keys easy to use 

1 = strongly agree 

5 = strongly disagree 

1.70 

.67 

1 Id. Tab Feature in Checklists easy to use 

1 = strongly agree 

5 = strongly disagree 

1.40 

.52 

1 le. Search Feature in NATOPS easy to 
use 

1 = strongly agree 

5 = strongly disagree 

1.70 

.95 

1 If. Performance Charts easy to use 

1 = strongly agree 

5 = strongly disagree 

1.50 

.53 

12. For IE-NATOPS: 




12a. Bookmark Function would be 
operationally useful 

1 = strongly agree 

5 = strongly disagree 

2.44 

.88 

12b. Performance Charts would be 
operationally useful 

1 = strongly agree 

5 = strongly disagree 

1.67 

.71 


3.4.1 General Comments from Pilots 


The following summarizes the comments made by pilots in completing the post¬ 
experiment questionnaire: 

Training 

• The usefulness of IE-NATOPS will be in the aircrew’s ability to quickly 
access emergency data. Scenarios should be developed that reflect this 
requirement. The scenarios used in this study were too complex. 

• Some scenarios asked for some data that would not be considered in an 
actual scenario. 


Confidence in Using IE-NATOPS 

• When using NATOPS (paper), one does not always have a specific location 
in mind when searching for a topic — one knows approximately where to 
find it. The IE-NATOPS makes it difficult to approximate. 

• The index and table of contents are extremely helpful. 

• A little more time and I would become very confident with the IE- 
NATOPS. 


NATOPS vs. IE-NATOPS 


• NATOPS is poorly organized, but still functions because of familiarity. 

• The paper PCL is difficult to use, tabs usually don’t open exactly to the 
section needed and the table of contents is not very user friendly. 

• IE-PCL is a big improvement. 

• Table of contents on paper NATOPS uses procedure names that are 
uncommon to most thought processes. 

• Resolution in some areas could be improved (IE-NATOPS). 

• The search and index functions are very, very effective at finding the 
relevant information. 

• The history, bookmark, and forward/back keys are well designed. 
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• History was upside-down — most recent should be at the top — didn’t 
see much need for the bookmark function. 

• The web browser needs more “intelligent” search functions. 

• Very good system, but should not completely replace NATOPS. Every 
pilot needs a printed NATOPS and PCL to highlight, study, etc. EE- 
NATOPS is a great backup to knowledge we should already have. 

• Performance charts should not have default values in them — once 
numbers are placed in the fields, the calculations should occur 
automatically. The current setup could lead to an error in reading old 
input if the enter button is not pushed. 

• Normal checklists scrolling into each other would be an improvement — 
scroll from section to section rather than going back to the menu. 

3.4.2 Correlational Analysis 

A correlational analysis was conducted on the questions from the survey (1-12) and the 
specific demographic variables provided in the biographical information questionnaire. 
The following correlations were found to be significant: 

1) Question 10b and flight hrs. in the H-60F [r(8) = .67, p = .034] 

2) Question 1 Id and flight hrs. in the H-60F [r(8) = .69, p = .028] 

3) Question 12a and age [r(7) = .70, p = .038] 

4) Question 1 and total flight time [r(8) = .64, p = .045] 

5) Question 2 and total flight time [r(8) = .64, p = .045] 

6) Question 3 and total flight time [r(8) = .64, p = .045] 

7) Question 9b and time in H-60F simulator [r(8) = .96, p = .000] 

8) Question 1 and age [r(8) = .65, p = .042] 

9) Question 2 and age [r(8) = .65, p = .042] 

10) Question 3 and age [r(8) = .65, p = .042] 

In summary, the above results can be loosely interpreted as meaning: 

• Pilots with more experience with the H-60F were less likely to rate IE- 
NATOPS capabilities/functions as positive as compared to pilots with 
less experience (1,2). 

• Older pilots with more overall flight time and/or simulator time were 
more likely to rate the IE-NATOPS training/scenarios as too complex, 
too regimented, and/or requiring too much practice time (4-6,7, and 8- 
10 ). 

3.5 Graphics Evaluation 
3.5.1 Technical Approach 

The approach taken in this effort started with developing an understanding of the tasks 
that pilots perform when using particular NATOPS graphics and schematics, studying 
how these graphics elements are used to support the tasks, and understanding how the 
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operational environment affects their use. Thus the initial step in the program was a 
series of interviews with pilots. This initial knowledge elicitation effort (see Deaton, 
Burke, & Good, 2000) was a significant driver in the conceptual design of early paper 
prototypes. These paper prototypes were presented to pilots in the current evaluation. 
This use of iterative design, where emphasis is placed on user feedback at multiple stages 
of design, supports the design of tools that will help pilots in the performance of tasks 
that require interaction with graphical figures and schematics in EE-NATOPS. 

3.5.2 Graphical Interaction Tool Interviews 

We conducted a series of interviews with pilots that participated in the IE-NATOPS 
benchmark evaluation described in this report. The purpose of these interviews was to 
get initial feedback on the graphical interaction tool concepts that we have developed (at 
Honeywell) in the course of this project. Each pilot was shown a variety of paper 
prototypes that described the concepts that we had developed to date. Each prototype 
was scaled down to mimic the size of the targeted display surface (approximately 6.4- 
inch diagonal screen). This was done to provide each pilot with a feeling for the size of 
the graphical images with which they would be dealing during flight. The interviewer 
described the functionality of each concept and feedback on the features of the concepts 
was solicited. The pilot was prompted to select the set of concepts that he felt would be 
the easiest with which to interact and that would provide the most utility during flight to 
accomplish the desired outcome. 

3.5.3 Graphical Interaction Tool Concepts 

This section describes the concepts that were presented to each pilot. Prior to this set of 
interviews, a detailed knowledge elicitation effort was conducted. For that effort, we 
interviewed several pilots at VX-1 (NAWCAD) to identify current airborne usage of 
NATOPS. Questionnaire and interview protocols were developed and data were obtained 
to identify and prioritize specific tasks accomplished with NATOPS. The effort also 
addressed information requirements associated specifically with the use of charts and 
other graphics within NATOPS. The results of this task suggest that performance charts 
and wind envelope charts are the most heavily used NATOPS graphics during flight. 
Therefore, the concepts we developed have focused on these two types of charts. 

3.5.3.1 Performance Charts. Figure 17 represents a potential shell for displaying 
performance charts. Note that this figure shows a performance chart at approximately the 
size it would appear on the targeted display surface for this program. We felt that it was 
important to provide an indication to the pilots of the size and resolution that would result 
from simply scaling a performance chart down to meet the display size requirements. The 
size of the image was quite small and extracting fine detail and specific point values out 
of the image would be difficult. It should be noted that the display surface of the system 
will be touch sensitive and the pilot will be interacting with information via a pen-like 
stylus. 
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Figure 17. Example of Scaled-down Performance Chart 


Concept PC 1 

At the top of the display the title of the particular chart is displayed along with other 
information specific to the chart. To the right of the chart title are two buttons labeled 
“CHART” and “NOTES”. When the CHART button is selected, the performance chart 
would be displayed. When the NOTES button is selected, a subsequent screen would be 
displayed that would contain other information specific to the chart. An example of a 
NOTES screen is presented in Figure 18. The information contained on the NOTES page 
is all the information found on the NATOPS version of the performance chart. To return 
to the performance chart view, the pilot simply selects CHART. 

Concept PC2 

It became apparent, after viewing a performance chart at the size of the targeted display, 
was that it might be useful to use some sort of highlighting or magnification to emphasize 
particular portions of the information contained in the chart. Our first set of concepts 
provides a “magnifying glass” or fisheye lens that appears when the pilot places the 
stylus on the display surface. Our first example of this technique is presented in Figure 
19. 
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Figure 18. The NOTES Page 



Figure 19. Magnifying Glass Concept 


In this concept, touching the stylus to the display surface causes a lens appears under the 
stylus location. This lens has the effect of magnifying the information below the tip of 
the stylus. One way to think about this is to imagine a glass ball that has been cut in half. 


35 



























By placing the flat side of the resulting hemisphere down on a performance chart, you 
create a crude magnifying glass. In addition to the magnification at the center of the lens 
(the “sweetspot”), the edges of the lens distort the image such that any particular line on 
the graph is connected on either side of the lens boundary. Thus, the user can maintain 
the distinction of which line is which since any particular line is still continuous across 
the lens boundary. The pilot is then able to move the stylus to any location on the chart 
and the lens will dynamically “follow” the stylus to highlight information at any location 
on the chart. Note that this particular performance chart does not necessarily lend itself 
to a magnification technique since the parameter curves within the chart are easily 
discemable. However, in situations where either the parameter curves converge or where 
many curves cross at the same point, it might be helpful to magnify those areas to provide 
a better understanding of what the parameter is doing in relation to the other variables 
displayed in the chart. Furthermore, since we are only at the concept development stage 
and are not certain that this concept will be useful to pilots, we have not resolved a 
number of factors that are relevant to this magnification concept. Factors such as the 
degree of magnification, the diameter of the “lens”, and the diameter of the “sweetspot” 
that would be most useful, as well as a host of other more general usability issues (e.g., 
highlighting color, ease of use, etc.) would need to be resolved if this concept is pursued 
in a development phase. 

Concent PC3 

While the concept of magnifying small portions of the chart may be useful in some 
situations, we considered other types of information on the charts that has utility for the 
pilots. The way a pilot typically uses a performance chart, which is generally represented 
as a type of nomogram, is to trace a line from either the x or y axis to a particular 
parameter curve and then down to the other axis to get the value for the variable in which 
they are interested (complex nomograms may require repetition of this process over 
multiple integrated axes). Based on this interaction, we extended the earlier concept to 
add axis guides and dynamic labeling of axis values. This concept is presented in Figure 
20 . 
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Figure 20. Axis Line/Magnifying Glass Concept 


As can be seen, this particular concept provides the magnified information, but also 
includes horizontal and vertical lines that extend to the axis. The axis lines and axis 
labels are dynamic and “follow” the location of the stylus. This particular concept allows 
the pilot to see fine details via the magnification lens if necessary, as well as the 
potentially more relevant axis information immediately. 


Concept PC4 

We noticed during the development of the previous concept was that it quickly became 
visually overwhelming for the user. In addition, most performance charts do not 
necessarily lend themselves to the need to access very detailed parameter information 
through a magnifying glass. That is, in most performance charts, the parameter curves 
within the chart are distinct and distinguishable. As can be seen in Figure 21, this 
concept provides a simpler view of performance chart parameters. Essentially, the only 
difference between this concept and the previous one is that the ability to obtain detailed, 
magnified information was eliminated. The functionality of the lines extending to each 


axis remains the same as before. 
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Figure 21. Axis Line Highlighting Concept 


Concept PC5 

The final performance chart concept that we presented to the pilots is shown in Figure 22. 
In this example, we maintain the idea of axis lines providing information to the pilot, but 
we have also included a different concept for including information about the parameter 
curves within the chart. As opposed to using a magnifying glass feature to get detailed 
information about the parameter, we have simply used a highlighting feature. In addition 
to the axis line feature described in other concepts, parameter curve information is also 
provided to the pilot. When the pilot moves the stylus, the axis line highlighting changes 
automatically to indicate the corresponding x and y values on each axis. Also, the 
parameter curve that is located most closely to the stylus is highlighted to give the pilot 
an easy and quick description of each parameter of interest within the chart. 
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Figure 22. Axis Line/Parameter Curve Highlighting Concept 


3.5.3.2 Wind Envelope Charts. The other type of graphical images that were noted as 
used quite frequently were wind envelopes. Wind envelope charts are presented in the 
PCL and do not appear in paper NATOPS manuals. A wind envelope chart is used to 
indicate the range of good/ideal wind conditions (speed and direction) that are acceptable 
during a hover over the ship. 

Each wind envelope is specific to a particular ship and includes wind envelopes for both 
day and night landing. Other parameters that are indicated on a wind envelope chart are 
RAST vs. non-RAST capable, port vs. starboard vs. stem approaches, and whether or not 
SAS boost is engaged or not. Wind envelope charts in the PCL also present day and night 
envelopes overlaid on each other on the same chart with shading being the only 
differentiating factor between them. It was difficult at first glance to determine which 
envelope corresponded to daytime landings and which corresponded to nighttime 
landings. Also, for a given ship type, the envelopes for the type of approach that was 
being taken appeared on different pages of the manual, thereby making it difficult at 
times to find the appropriate envelope that was required. The concept that we have 
developed integrates these factors into one display and allows the pilot to simply select a 
set of parameters that control the type of envelope that will be displayed. 
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Concept WE 1 

In Figure 23, you can see an outline of the ship in addition to a “parameter selection” 
area. The pilot would select the “Ship Type” via a drop-down menu that would include 
information for each type of ship that is listed in the PCL. The pilot would also select 
specific parameters from the parameter selection area that correspond to the type of 
information they desire. These parameters control the graphical representation of the 
figure on the left side of the display. Once the parameters are selected, the figure would 
be automatically updated with the information supplied by the pilot and the correct 
envelope would be displayed. 
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Figure 23. Wind-Envelope Chart Concept 

3.5.4 Interview Feedback and Recommendations 

The pilots were asked to comment on particular features of each of the concepts, 
indicating things that were useful and also identifying those features that were not useful. 
They were then asked to focus on to a particular concept or small set of concepts that 
they felt would provide them with the easiest access to the types of information that they 
need. For this discussion, we will break the responses into categories relating to 
performance charts and those relating to wind envelope charts. 

Performance Charts 

Every pilot felt that the best, and most useful, concept was Concept PC5 (axis lines plus 
parameter curve highlighting). They felt that this type of display provided them with the 
all of the values that are of interest to them when they are interacting with a performance 
chart. Each pilot liked the idea that all important parameters are quickly available and 
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felt that the feedback provided to them through the use of highlighting would allow them 
to immediately determine the particular value for which they were looking. 

Some pilots felt that it was unnecessary to highlight the entire parameter curve for 
Concept PC5. They felt that you might be able to simply drop a digital value of the 
parameter curve off from the tip of the stylus. An example of this is displayed below in 
Figure 22. 

Some pilots felt that the magnification concepts, particularly PC3 (magnification with 
axis line highlighting) could be useful in situations where you need precise information 
about the parameter curves. In addition, some pilots felt that, for some charts, having a 
magnification lens might make it much easier to differentiate between lines, especially 
when the parameter curves converge to a common location. However, even the pilots 
that liked the magnification lens concept felt that if it were included in a graphics 
interaction tool, it should be a selectable feature and not something that would have to be 
used continually. 

One of the factors that was mentioned as a disadvantage for the magnification lens 
concepts was related to the distortion that appears at the edges of the lens. Upon a quick 
glance at the chart, pilots felt it might be difficult to “follow” the parameter lines as they 
become distorted toward the edge of the lens based on the algorithms that were used to 
construct the magnification. This would be an issue that would need to be resolved if the 
magnification lens concept is pursued. 

It was noted that the use of color is an important issue. There is no color standard that is 
currently used in these aircraft; however, pilots frequently use night-vision goggles and 
any color scheme must be conducive to use with these systems. 

Some of the pilots liked the variety of features that were presented in these concepts. It 
was suggested that we might include a “display management configuration” menu that 
would include selectable parameters for chart interaction. This might include the ability 
to turn on and off features like magnification, axis highlighting, parameter curve 
highlighting, etc. 

Based on the feedback from the pilots and the scope of the project, the most widely 
accepted concept was chosen (Concept PC5) for further development and was modified 
to accommodate some of the recommendations that were provided by the pilots. The 
resulting interface is presented below in Figure 24. 
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Figure 24. Recommendation for Performance Chart Graphical Interaction Concept 


Wind Envelope Charts 

Every pilot felt that the wind envelope concept, as it was presented, would be very useful 
during flight. 

It was recommended that salient text information be presented at the top of the chart on 
the chart view for reference purposes to indicate the type of ship and the deck-type on 
which the aircraft was landing. 

All pilots mentioned that “STERN” should be included in the parameter selection portion 
of the display to provide wind envelope information for Stem approaches to the ship. 

It was noted that there should be some intelligence built into the interface. For example, 
some ships are only Non-RAST capable and for those types of ships the parameter should 
simply default to “Non-RAST.” 

There were some recommendations about possible redesigns of the interface. Many of 
these centered on desirable feedback to present to the pilot about current parameter 
selections. It was suggested that, instead of a radio button technique for the pilot to select 
particular parameters, an alternative drop-down menu containing the possible selections 
be used instead. This would provide immediate feedback to the pilot about the particular 
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set of selected parameters that would be preferable to relying solely on whether the circle 
next to the parameter was filled (selected) or not (not selected). 


Based on the recommendations from the pilots, the wind envelope interface was 
redesigned and the resulting concept is presented below in Figure 25. 
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Figure 25. Recommendation for Wind Envelope Graphical Interaction Concept 
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4. Discussion 

4.1 Main Effects 


The benchmark evaluation experiments were conducted to provide an initial assessment 
of whether aviator performance would differ dependent on the search tool that was used - 
traditional paper NATOPS/PCL or an electronic version of these tools (IE-NATOPS). 

The results from these experiments provide preliminary evidence that aviator 
performance may differ dependent on the search mode, as well as the exact purpose of 
the search. For example, within experiment 1, content-access trials measured the 
participants’ other-directed access of information in the paper NATOPS/PCL and the IE- 
NATOPS. The participants knew where they were starting and what they had to find in 
these documents, and they simply had to navigate to the designated information. Under 
these conditions, results indicate that access times to reach a given page of information 
were significantly faster for IE-NATOPS than for the paper NATOPS/PCL. This 
improvement in other-directed access time with IE-NATOPS over the paper NATOPS 
and PCL suggests that, when participants know what kind of information they are 
seeking, IE-NATOPS is a faster means to access that information than the paper 
NATOPS/PCL. 

A slightly different picture was seen in the second study where self-direct search rates 
were examined. In experiment 2, problem-solving trials were used to measure the 
participants’ self-directed search time and accuracy for resolving problem scenarios using 
the paper NATOPS/PCL and IE-NATOPS. Results indicate that, overall, there was not a 
significant difference in self-directed search time and accuracy between the paper 
NATOPS/PCL and IE-NATOPS. This null difference between NATOPS/PCL formats 
during the self-directed search process implies that the IE-NATOPS format of the 
NATOPS and PCL may provide equivalent support for situations in which aviators know 
the general section they must access, but do not know the specific sections of the paper 
format. For within these situations, search times are tapping not only search rates, but are 
also dependent on the variations in search strategies that individuals may adopt. 

Although the studies contained within the benchmark evaluation were not intended to 
provide specific information of how context may impact which search tool is most 
effective, they may provide a clue. Sequential search problems within the study 
examining access time were designed to be non-equivalent in that they grew 
progressively more complex to ensure that participants had an adequate knowledge of 
how to manipulate the interface before moving on to more difficult cases. As the specific 
effect of individual searches was not the focus of the current benchmark, no specific 
hypotheses were made in this regard. Post-hoc analyses indicated that there were 
differences among the scenarios, with most favoring IE-NATOPS, but we believe that the 
differences found were primarily in the complexity of the searches. 

Problem solving scenarios, within experiment 2, were originally intended to sample a 
variety of information types and be uniform in their complexity. A panel of SMEs who 
reviewed these scenarios before the data collection effort took place felt that the 
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complexity of the scenarios was uniform across the set. The experimenters' observations 
during data collection, however, revealed that there seemed to be some scenarios that 
caused participants to have longer decision and access times than other scenarios. 
Additionally, there seemed to be some characteristics of the scenarios that consistently 
favored either the paper-based NATOPS/PCL or IE-NATOPS. For example, when 
participants were required to use performance charts, search times/problem solutions 
were quicker using the electronic search tool (DE-NATOPS). In addition, when 
completing checklists where hyperlinks were inserted in IE-NATOPS, the electronic 
search tool seemed to produce quicker search times. Conversely, it appears that scenarios 
that required participants to search for diagrams or charts lent themselves to the paper 
search tool rather than the electronic version. In many instances it appeared that 
participants, while knowing the general section where required information could be 
found, did not know the specific subsections. The paper search tool would seem to make 
it easier to find information in this situation (easier to scan), especially something that 
would stand out like a chart or graphic. 

Given that the aviators who participated in this benchmark evaluation of IE-NATOPS 
had less than one hour of training on IE-NATOPS, and given that the aviators were 
highly experienced in the use of the paper NATOPS and PCL for accessing information 
to resolve problem scenarios, the results from the content access trials and problem 
solving trials are indicative of a system that may prove to be superior to the existing 
method of accessing NATOPS data in the aircraft. Since the IE-NATOPS used for this 
benchmark evaluation is a prototype system, there is the potential for further 
improvements in the interface, possibly resulting in improvements in access times - 
which are already faster than the paper NATOPS and PCL - and decision support. 

Additionally, though this benchmark evaluation was not intended to do so, the results of 
this study suggest that there may be characteristics of scenarios that may favor the paper- 
based NATOPS and PCL information, the current version of IE-NATOPS, or have no 
format bias. Although results/trends of the current study may be argued to suggest that 
scenario/situational characteristics may impact the gains from an electronic versus paper 
search tool, the limitation of the current study is that it was not designed to examine this 
question. Therefore, any noted trends are based on observation, combined with a post- 
hoc analysis of scenario content. Despite this limitation, this implication is significant in 
that, as far as the authors are aware, there is no research in the published literature that 
has addressed the characteristics of scenarios and their interaction with paper versus 
electronic forms of NATOPS and PCL. If NATOPS and PCL data displays prove to be 
similar in terms of human performance to the moving map display, it may be possible to 
eliminate the bias of one format over the other by development and application of 
interface elements. A study designed to manipulate the characteristics of scenarios and 
their interaction with the information format and participant knowledge and experience is 
necessary to establish these characteristics and associated interactions. Once these 
characteristics and interactions are established, the results may be applied to the design of 
the NATOPS and PCL interfaces to negate any biases in the majority of scenario 
characteristics and user experience. 
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4.2 Study Limitations 


A potential limitation of this exploration of scenario characteristics with the current study 
is that the participants were all aviators with extensive experience using the paper 
NATOPS and PCL. In addition, the participants’ experience with the specific 
information types and content was not controlled. Likewise, the scenarios were not 
designed to vary specific information types and content with interface type. The 
limitations of the benchmark evaluation introduce sources of variability that remain 
unaccounted for by the design of the current study, affecting the ability to unambiguously 
establish the scenario characteristics favoring one or both of the NATOPS and PCL 
interface types. These limitations do not affect the primary purpose of this benchmark 
evaluation, however. The extemal/other and internal/self directed access times for IE- 
NATOPS and their comparison to the paper-based NATOPS and PCL were clearly 
established. 

4.3 Graphics Evaluation 

The scenarios in the benchmark evaluation did not allow for systematic variation of the 
electronic graphical images like charts of schematics that the participating pilots used 
during their data searches. To address interaction issues with electronic graphics we 
interviewed the pilots after the benchmark evaluation. As mentioned earlier, we 
presented a variety of interaction methodologies to the pilots for interacting with 
performance charts and simply solicited feedback from each pilot on each of the 
concepts. In general, the results of the interviews suggested that pilots felt the techniques 
used when interacting with graphics are very important to the usefulness of the IE- 
NATOPS system. Due to the small size of the targeted IE-NATOPS display, pilots felt 
that the ways in which they interacted with graphics in the paper NATOPS manual would 
not be adequate when a performance chart was reduced in size. More specifically, every 
pilot felt that the best, and most useful, concept was dynamic labeling. They felt that this 
type of display provided them with all of the values that are of interest when they are 
interacting with a performance chart. Each pilot liked the fact that all important 
parameters are quickly available and felt that the feedback provided to them through the 
use of highlighting and labeling would allow them to immediately determine the 
particular value for which they were looking. However, some pilots felt that it was 
unnecessary to highlight the entire parameter curve. Some suggested that the value of the 
parameter simply be displayed off the tip of the stylus to limit the spread of information 
within the display. 

Some pilots felt that dynamic magnification could be useful in situations where you need 
precise information about the parameter curves or where parameter curves converge to a 
single point. However, the pilots mentioned that typically they use performance charts to 
“eye-ball” particular trends or system performance information. Because of this, many of 
the pilots felt that such a feature would result in a display that was too cluttered and 
somewhat difficult to use. Some of the pilots, on the other hand, felt that for specific 
types of charts (e.g., complicated charts with many parameters), there might be some 
utility of dynamic magnification. In future designs of the graphical interaction tools, this 
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could be a selectable feature and not necessarily always present during interactions with 
the performance chart. We plan to use the information gathered in this evaluation to 
further refine the concepts that were presented to the pilots in this test. 

4.4 The Future 


4.4.1. Future Research 

While the benchmark evaluation of the prototype IE-NATOPS indicated that the potential 
is there for IE-NATOPS to produce quicker access times, as well as processing times, the 
strength of these relationships seemed to vary, depending on the characteristics of 
targeted scenarios. As the current study was not designed to examine the effect of search 
mode on scenario characteristics, results regarding specific scenario effects are post-hoc 
and must be taken in light of the limitations imposed with respect to the specific scenario 
effects that have been noted. Future research should design empirical studies to 
systematically investigate what characteristics might favor the use of IE-NATOPS, as 
compared to traditional NATOPS/PCL. In addition, based on the results of the current 
study and comments from the aviators regarding the IE-NATOPS prototype system, the 
concept will be accordingly revised and improved. These future revisions should be 
empirically tested. 

Regarding the graphical interaction techniques described in this paper, the recent 
proliferation of small handheld devices and PDAs suggests that it is imperative that basic 
research continue in the identification of novel and useful interaction methodologies for 
situations when large information spaces are presented on a small display surface. It is 
critical that future research continue this iterative approach (concept design, testing, 
redesign) so that the graphical interaction methodologies become intuitive, easy-to-use, 
and increasingly supportive of the needs of the pilot community. 

4.4.2. Future Modifications/Directions 

The findings from the benchmark evaluation have provided many areas in which the 
design/interface of IE-NATOPS may be able to be modified such that the benefits of this 
tool are further enhanced. The primary source of suggested changes in the functionality 
or interface portions of IE-NATOPS came from the aviators who used the system in the 
benchmark evaluation (see section 3.4.1), although experimenters also noted a few 
suggestions based on observing aviator interaction with the system. 

One example of a possible improvement to the design of the IE-NATOPS interface that 
could further enhance access times - particularly when self-directed - and possibly 
enhance decision-making times over and above those of paper NATOPS and PCL 
involves the method of accessing subsections in each section of NATOPS and PCL. In 
the version of IE-NATOPS employed in this study, participants had to select each 
individual subsection they wished to view. The participants had to back out of a 
subsection, then select the next subsection they wished to view. When the participants 
did not know the exact location of an item, this method of navigating IE-NATOPS may 
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have resulted in poorer search times than were possible with the paper NATOPS and 
PCL. Thus, by allowing the participants to select and/or scroll through the subsections of 
IE-NATOPS, search times could be improved. Using this scrolling method, IE-NATOPS 
would be like a web page that uses hyperlinks (the IE-NATOPS index) and hypertext for 
sections of the page, allowing the user to jump to a section of the page and then scroll 
backwards and forwards starting at that location. 

Another possible modification might involve the following. Since a paper version of 
NATOPS and the PCL will be carried in aircraft for many years to come, it is imperative 
that IE-NATOPS and paper NATOPS and PCL should use similar organization and 
navigation cues for information contained therein. Accordingly, a comprehensive index 
that is meaningful (to the users ) and with accurately placed tabs or other quick-reference 
cues, and a consistent layout and means of control of information across sections should 
be employed. 

IE-NATOPS has potential uses not only as a decision-aid while in the cockpit, but also as 
a training tool that can be implemented in support of current NATOPS training 
procedures. As the tool is electronic, it represents a more portable training reference as 
well as a method to support delivery of training via the design of event-based scenarios in 
which certain competencies dealing with IE-NATOPS are targeted. This would allow 
targeted practice and feedback to be built into the system to augment the feedback given 
by the instructor(s). To pursue such training applications, a targeted analysis could be 
conducted of current NATOPS training procedures to: (a) identify strengths and 
weaknesses of the current method, (b) determine where IE-NATOPS may fit into the 
training rotation, and (c) the identify the competencies that should be addressed with this 
tool. 

Further IE-NATOPS development efforts are warranted in both operational development 
and research. Operational development must be conducted to resolve issues in the areas 
of institutionalization, aircraft integration, and interface design. Institutionalization 
issues will include coordination with NATOPS committee members and the Naval Air 
Systems program office for training systems (PMA 205) to identify concept issues related 
to training and to integration of the IE-NATOPS into the NATOPS creation and update 
process. Flight evaluations of a certified IE-NATOPS system will be required to address 
aircraft integration and interface design issues. Future research efforts will be required to 
develop HUMS diagnosis logic for cockpit alerting and tools to interact with graphics on 
small screens. Research can also be conducted to investigate alternative tools for 
performance data calculations and checklist interface features. 
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Appendix A. Participant Briefing 


Participant Briefing: IE-NATOPS Benchmark Evaluation Effort 

The goal of this project is to investigate an interface concept for an interactive electronic 
NATOPS (IE-NATOPS) to support aircrew performance. This study represents a 
benchmark evaluation of the prototype IE-NATOPS. This study will compare aircrew 
performance when using the traditional paper copy of NATOPS and PCL with that of an 
electronic version presented via a computer interface. Results from this study will be 
used to refine the IE-NATOPS concept. 

If you choose to participate in this experiment today, you will be introduced to the IE- 
NATOPS interface and given the opportunity to practice using it. You will then be asked 
to work through a number of scenarios in which the NATOPS and PCL, or IE-NATOPS 
will be made available for use. Specifically, you will be asked to resolve a number of 
problems in each scenario using the different versions of NATOPS. We will be 
collecting data on how you go about resolving the problem scenarios, how long it takes to 
reach a solution, and the outcome of your solution. 

There are no foreseeable risks to you by participating in this research. At times, you may 
feel a small amount of stress similar to what you may have felt during your annual 
NATOPS test. All data are collected anonymously, such that there is no way of 
identifying a particular participant with his or her scores or responses on the 
questionnaires. 

Your participation in this experiment is voluntary. You may terminate your participation 
at any time without any adverse consequences. If you need any additional information 
about this experiment, please contact: 

Randall L. Oser 

Naval Air Warfare Center Training Systems Division, AIR 4961 
Orlando, Florida 32826-3275 
(407) 380-4818 
OserRL@navair.navy.mil 
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Appendix B. Privacy Act Statement 


Privacy Act Statement 


1. Authority . 5 U.S.C. 301 

2. Purpose . The use of NATOPS and PCL by naval aviators to resolve paper-based 
problem scenarios will be evaluated using methods outlined in the study titled: 
"Condition Based Maintenance: Benchmark evaluation effort supporting development 
of an electronic NATOPS manual for the H-60." This study is to be performed in an 
attempt to understand if differences exist in the processes used by naval aviators 
when using traditional paper-based NATOPS and PCL with that of an electronic 
version currently implemented by the Interactive Electronic NATOPS project (LE- 
NATOPS). 

3. Routine Uses . The data collected will be used for analyses and reports by the 
Departments of the Navy and Defense, other U.S. Government agencies, and 
authorized government contractors. Additional use of the information may be granted 
to non-Govemment agencies or individuals by the Navy Surgeon General following 
the provisions of the Freedom of Information Act or contracts and agreements. I 
voluntarily agree to its disclosure to the agencies or individuals identified above, and 
I have been informed that failure to agree to this disclosure may make the research 
less useful. 
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Appendix C. Informed Voluntary Consent to Participate 

Informed Voluntary Consent To Participate: 

IE-NATOPS Benchmark Evaluation Effort 

1. I am being asked to voluntarily participate in a data collection effort titled. 
Benchmark evaluation effort supporting development of an interactive electronic 
NATOPS manual for the H-60. 

2. During the research, I will resolve paper-based problem scenarios using paper- 
based and electronic NATOPS and PCL, and participate in a short de-brief on my 
experiences subsequent to the session. Portions of this session may be audio- 
taped to assist in later data analysis. However, all collected information will 
remain confidential and anonymous and will not impact you in any way. 

3. I understand that the investigators believe that the risks or discomforts to me are 
as follows: 

• No greater than would be experienced during the participation in a typical 
School house class 

• No greater than would be experienced during responding to a typical Navy 
survey. 

• No greater than would be experienced while performing my annual NATOPS 
test. 

4. The benefits that I may expect from my participation in this study are minimal. I 
understand that I will receive no direct benefit other than the knowledge that 
participation in this study will aid efforts to improve the performance, safety, 
and/or effectiveness of the US Navy. 

5. My confidentiality during the study will be ensured by assigning me a coded 
identification number. My name will not be directly associated with any data. The 
confidentiality of the information related to my participation in this research will 
be ensured by maintaining records only coded by identification numbers. Video 
and photographic images of me will not be published or displayed without my 
specific written permission. 

6. If I have questions about this study I should contact the following individuals: 

Randall L. Oser 

Naval Air Warfare Center Training Systems Division, AIR 4961 
Orlando, Florida 32826-3275 
(407) 380-4818 
OserRL@navair.navy.mil 

7. My participation in this study is completely voluntary. 
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8. No additional out of pocket costs to me may result from my voluntary 
participation in this study. 

9. Official government agencies, such as the Naval Bureau of Medicine and Surgery, 
may have a need to inspect the research records from this study, including mine, 
in order to fulfill their responsibilities. 

10.1 have received a statement informing me about the provisions of the Privacy Act. 

11.1 have been informed that the Process Administrator is responsible for storage of 
research records related to my participation in this data collection effort. My 
consent form will be stored under lock and key in compliance with NAWCTSD 
Instruction 3900, Protection of Human Subjects (Pending). 

12.1 have been given an opportunity to ask questions about this study and its related 
procedures and risks, as well as any of the other information contained in this 
consent form. All my questions have been answered to my satisfaction. I 
understand what has been explained in this consent form about my participation in 
this study. I do not need any further information to make a decision whether or 
not to volunteer as a participant in this study. By my signature below, I give my 
voluntary informed consent to participate in the research as it has been explained 
to me, and I acknowledge receipt of a copy of this form for my own personal 
records. 


Volunteer Signature 


Name SSN Date 


Principal Investigator Signature 


Name SSN Date 
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Appendix D. Biographical Information 

Biographical Information 

Rank/Rate: _ 

Sex: M_ F_ 

Age: _ 

Squadron: _Time in squadron: _(months) 

Time in Service (months): _ # of Deployments: _ 

Current Assignment (e.g., OPS, OPSO, SKEDS): _ 

Past Assignments: 

Special Qualifications (e.g., NATOPS Instructor, ASST NATOPS Instructor, FCP, 
SWTI, Instructor duty, 

Test Pilot, etc.): 


Time in Present Assignment:_(months) 

Time in Previous Assignment: _(months) 

Aircraft Hours (approx.) Role ' Aircraft Hours (approx.) Role 1 


HH-60F 


Approx, time in H-60F over: 

Last 30 days: _ Last 60 days: _ Last 90 

days: _ 

Approx, time in H-60H over: 

Last 30 days: _ Last 60 days: _ Last 90 

days: _ 
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Approx, total time in H-60F Simulator: _(hours) 

Total Time in Rotary Wing: _(hours) 

Total Flight Time: _(hours) 

Approx, hours of use of desktop/laptop personal computers per week: _ 

Approx, hours of use of PDA or other handheld computing device per week: 
_(hours) 

1 : e.g., indicate whether hours were obtained as helicopter commander. 


.(hours) 
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Appendix E. Post-Experiment Questionnaire 

POST-EXPERIMENT QUESTIONNAIRE 


The purpose of this questionnaire is to provide you with a means to comment on issues 
regarding the IE-NATOPS and the experiment in which you have just participated. As 
with the data collected during the experiment, your comments on this questionnaire will 
remain confidential. 

This questionnaire contains 12 questions. Select your response from the response items 
provided. Make your selection by circling the letter for that selection. If you would like 
to elaborate on any of your responses, you may do so in the space provided. If you have 
any questions regarding any items in this questionnaire, feel free to ask the experimenter 
for clarification. 


1. IE-NATOPS training organization was... 

(a) ...too loose. I needed more structure in the training program. 

(b) ...perfect for my needs. 

(c) ...too regimented. I needed to be able to control the pace and direction of my 
study more than I was allowed. 

Comments: 


2. IE-NATOPS training contained... 

(a) ...little I needed to learn to use the system to resolve the problem scenarios. 

(b) ...sufficient content to learn to use the system to resolve the problem 
scenarios. 

(c) ...more than I needed to learn to use the system to resolve the problem 
scenarios. 

Comments: 


3. IE-NATOPS training contained... 

(a) ...too little practice time. 

(b) ...sufficient practice time. 

(c) ...too much practice time 
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Comments: 


4. IE-NATOPS training scenarios were... 

(a) ...tooeasy. 

(b) ...sufficiently complex. 

(c) ...toocomplex. 

Comments: 


5. Confidence regarding your use of IE-NATOPS for acquiring information in flight: 

(a) Very confident. I can go directly to information I need to access in the system 
without having to search for the item(s). 

(b) Confident. I can find information I need to access in the system in a timely 

manner. 

(c) Uncertain. I might be able to find information I need to access in the system 
in a timely manner. 

(d) Very uncertain. I probably won't be able to find information I need to access 
in the system in a timely manner. 

Comments: 


6. The interface was physically easy to manipulate. 


NATOPS 


IE-NATOPS 


(a) Strongly agree 

(b) Agree 

(c) Neutral 

(d) Disagree 

(e) Strongly disagree 

Comments: 


(a) Strongly agree 

(b) Agree 

(c) Neutral 

(d) Disagree 

(e) Strongly disagree 


7. I never had a doubt as to what I had to do to the interface to access data. 
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NATOPS 

(a) Strongly agree 

(b) Agree 

(c) Neutral 

(d) Disagree 

(e) Strongly disagree 

Comments: 


IE-NATOPS 

(a) Strongly agree 

(b) Agree 

(c) Neutral 

(d) Disagree 

(e) Strongly disagree 


8. I had no trouble reading and understanding the NATOPS data. 


NATOPS 

(a) Strongly agree 

(b) Agree 

(c) Neutral 

(d) Disagree 

(e) Strongly disagree 


IE-NATOPS 

(a) Strongly agree 

(b) Agree 

(c) Neutral 

(d) Disagree 

(e) Strongly disagree 


Comments: 


9. I had no trouble finding data I needed to use to resolve the scenarios. 


NATOPS 


IE-NATOPS 


(a) Strongly agree 

(b) Agree 

(c) Neutral 

(d) Disagree 

(e) Strongly disagree 

Comments: 


(a) Strongly agree 

(b) Agree 

(c) Neutral 

(d) Disagree 

(e) Strongly disagree 
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10. It was easy to access information that involved searching in multiple sections. 


NATOPS 

(a) Strongly agree 

(b) Agree 

(c) Neutral 

(d) Disagree 

(e) Strongly disagree 


IE-NATOPS 

(a) Strongly agree 

(b) Agree 

(c) Neutral 

(d) Disagree 

(e) Strongly disagree 


Comments: 


11. For each of the following aspects of the EE-NATOPS please indicate your agreement 
with the following statement regarding their ease of use. Ease of use is defined as 
the degree to which it was easy to manipulate the interface to find the information 
that was needed. Indicate your level of agreement by putting an “x” in the 
appropriate box. 


“I found that this function was easy to use.” 



Strongly 

Agree 




Strongly 

Disagree 

History Function 






Bookmark Function 






Navigation Keys 
(Forward/Backward) 






Tab Feature Within 
Checklist Section 






Search Feature Within 
NATOPS Section 






Performance Charts 







Comments: 
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12. For each of the following aspects of the IE-NATOPS please indicate your agreement 
with the following statement. Indicate your level of agreement by putting an “x” in 
the appropriate box. 

“I would expect this function to be functional (useful) within an operational 
environment (in the aircraft).” 



Strongly 

Agree 

Agree 

Neutral 

Disagree 

Strongly 

Disagree 

Bookmark Function 






Performance Charts 







Comments: 
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Appendix F. Participant Debriefing 

PARTICIPANT DEBRIEFING 

The study in which you have just participated is being performed in an attempt to 
understand if differences exist in the processes used by naval aviators when using 
traditional paper-based NATOPS and PCL compared to a prototype interaction electronic 
NATOPS (IE-NATOPS). The study compares aircrew performance when using the 
traditional paper copy of NATOPS and PCL with that of an electronic version presented 
on a computer interface. Results from this study will be used to refine the IE-NATOPS 
concept. 

The purpose of the IE-NATOPS project is to facilitate the in-flight decision-making 
processes requiring NATOPS data. By making improvements to the organization, 
presentation, and accessibility of the NATOPS data, it is anticipated that more timely 
resolution of systems and performance related problems occurring in flight may result 
than is observed with the current paper-based NATOPS format. 

This study represents a benchmark evaluation of the prototype IE-NATOPS. The study is 
a continuation of the work that has been conducted in developing an electronic format for 
NATOPS in the H-60 rotorcraft. The major players in this effort have included the Naval 
Air Warfare Center Training Systems Division (NAWCTSD), the Office of Naval 
Research (ONR), the University of Central Florida, CHI Systems, Inc., and Honeywell 
Technology Center. 

If you have any further questions, or if you would like to learn the results of this study 
upon its completion, please contact: 

Randall L. Oser 

Naval Air Warfare Center Training Systems Division, AIR 4961 
Orlando, FL 32826-3275 
(407) 380-4818 
OserRL@navair.navy.mil 
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Appendix G. IE-NATOPS Training 

G.1 IE-NATOPS Training Protocol 


IE-NATOPS TRAINING (Do not read) 

The next section of the study will require you to utilize an electronic version of 
NATOPS. However, prior to this we are going to conduct a short training session so that 
you feel comfortable operating this electronic version. Training will proceed in the 
following manner. First, I will briefly explain the general features of the IE-NATOPS 
(basically the layout of the initial screen). Then I will progress to describing the features 
of each of the main six modes (checklist, trends, data, alert, NATOPS, notes) as well as 
how to navigate within these modes. Finally, we will conclude with several hands on 
practice sessions in which you will use the IE-NATOPS to locate various information. 
The information that you will be required to find within these practice sessions will be 
similar to the information that the paper based scenarios will require you to access within 
the actual study. This process will be highly interactive so if at any time you have 
questions, please feel free to ask. 

I GENERAL INFORMATION (DO NOT READ) 

In many ways the electronic version (IE-NATOPS) is configured similarly to the paper 
NATOPS in terms of organization of content. The main differences are: (1) many 
sections contain hyperlinks (identified by blue text, clicking on it will take you to another 
section - replace “go to” sections in paper NATOPS, and (2) that the navigation within 
sections (e.g., checklist, trends, data, alert, NATOPS, notes) is most akin to that of an 
internet browser. 

Instructor: At this point pull up IE-NATOPS... 

We’ll start with a general overview, hitting the high points concerning general 
navigation capabilities. Most of the navigation features will cut across the various 
aspects (checklists, NATOPS, performance data, etc..) of the IE-NATOPS and as 
such will be further elaborated upon and practiced once we begin talking about 
each of the five sections of the IE-NATOPS. 

Instructor: Interactively walk through each of the following points describing the 
general layout of the first screen accessed... 

General Navigation 

• First row of blue buttons (point) provide navigation through each of 
the 6 modes corresponding to checklists, trends, data, alerts, NATOPS, 
and notes. Only four are activated for use in the current version 
(checklist, data, NATOPS, and notes). To select a button click on it 
and the button will turn yellow to let you know its activated/selected. 
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Give the following quick explanation of each of the 6 modes as you 
click on it to show them that active button becomes yellow. 

o Checklist: Corresponds to information in PCL. No foldouts. 
o Trends: Not activated within this version, 
o Data: Calculation of performance information (done 
automatically versus how do it now - by hand) 
o Alerts: Not activated within this version, 
o NATOPS: Corresponds to information contained in paper 
NATOPS. No foldout charts... 
o Notes: Electronic notepad, probably not that useful within 
scenarios will be given today. 

• Second row of blue buttons allow for navigation within (and between) 
screens, similar to an internet browser. Not all of the buttons are 
available for use within each mode (mode and situation dependent); if 
unavailable the text on the button will tend to gray out/unbold. 

o For example, Fwd and PCL buttons currently grayed out. 

Looking at the navigation tools, generally the following navigational 
tools are available within all modes: Back/Fwd, History, Bookmarks, 
the last button varies dependent on mode currently operating in (will 
be discussed in detail later). Illustrate each. 

o Back/Fwd: Operates same as back/fwd button on web browser. 
Each time you hit the back button it takes you back a step, 
while forward takes you forward. Notice that at this point the 
forward button is grayed out because we just started, 
o History: Let’s you track your path of where you have been 
(e.g., what you have accessed thus far). Clicking on any item 
in the list will take you back to that section. Only contains a 
maximum of ten items, older items kicked out. 
o Bookmark: Operates same as does a bookmark within 
web/intemet applications. Allows you to set bookmarks for 
frequently visited places. Again once bookmark set, click on it 
and it will take you back to that topic, 
o The last button on the right will change depending on the mode 
that you are currently in so we will wait and talk about that 
when we get into detail on each mode. 

• The next large set of buttons allows the user to navigate through the 
information unique to each mode. In the checklist mode these larger 
buttons are yellow, however within NATOPS mode they are arranged 
like a Table of Contents. In either case if you click on one of these 
buttons a tree diagram will be revealed that further expands on the 
original topic. Once you begin to do this you will see a tree diagram 
that has either a “+” or next to the topic of interest. Those will a 


64 




“+” can be further expanded by clicking on the “+” symbol. We’ll 
play with this a little more later. 

• Two smaller blue buttons on the bottom allow you to configure each 
screen when applicable. For example, it allows you to change the 
color of charts within the trend mode, modify screen for default 
configuration according for day, night, or NVG flights. The second 
button is an on-line help button. The on-line help button is not 
completed within this version and for the scenarios today you won’t be 
required to use the configure button. Just wanted to give you a brief 
description. 


II MODE SPECIFIC IE-NATOPS TRAINING (DO NOT READ) 
CHECKLIST MODE 


Function(s) 

• The checklist mode is the default mode upon start-up and corresponds 
to the information contained within the PCL. As the checklist mode is 
currently active you will note that this button is currently yellow. 

• As the checklist mode mimics the PCL, you will also see five blue 
buttons toward the bottom of the screen, corresponding to each of the 
major PCL sections/tabs (EP, NP, Special, Reference, Functional). 
Note functional checklist is not available within this version - grayed 
out. 

• Furthermore, as the EP section is the default mode on start-up the EP 
button at the bottom of the screen appears depressed to indicate its 
active (look closely active button is surrounded by dotted lines). Single 
click on tab brings each of other sections up to front. Only one section 
is available at a time. 

• Once a particular mode is activated the second row of buttons will 
indicate what navigational tools may be utilized within this mode. So 
within the checklist mode the following buttons are operational. 

Those not grayed out are available. 

o Back/Fwd: Same function as before ( illustrate ) 
o History: Same function as before ( illustrate) 
o Bookmark: Same function as before ( illustrate ) 
o PCL: If active, indicates that there is at least one checklist 
that is still open and has not been fully completed. Checklist 
is fully complete when you have clicked upon the complete 
button at the bottom. Clicking on PCL will illustrate each 
open checklist. More later. 

Getting to the Actual Checklist 
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• Large yellow buttons below (indicate) illustrate the first layer of 
information available within the checklist mode. Should correspond to 
similar sections of PCL. Clicking on the large yellow buttons will 
provide the user with a subset of checklists relevant to the category 
clicked upon within the main menu. Once you begin to do this you 
will see a tree diagram that has either a “+” or next to the topic of 
interest. Those will a “+” can be further expanded by clicking on the 
“+” symbol. Those with a are already expanded fully. Once 
checklists are fully expanded click on the specific checklist needed and 
it will appear. 

• As you are progressing through a checklist can click on each step and 
a box will appear around it, letting you know that that particular step 
has been completed (good for keeping track of where currently are in 
case of disruption). Once you reach the end of the checklist it is 
important that you hit the “completed” button as this will close the 
checklist and kick you back into the next higher menu. Illustrate this. 

• If you do not fully complete a checklist, but move on to another area 
the PCL navigational button will become active. The PCL button 
indicates that a checklist is still open. Clicking on this button will 
allow the user to see and access all checklists currently open (each will 
have its own tab, active one in yellow). Illustrate. 

• Multiple checklists can be opened at once and each will have its own 
tab. User can switch between checklists by clicking on desired tab; 
active remains yellow, non-active blue. 

Moving around within checklists 

• Scroll button on right. For longer checklists you will notice a scroll 
button on the right. To scroll up or down can either click on the arrow 
at either end of the scroll button or can click and drag the gray bar at 
the right (bar in between the arrows at either end of scroll bar). To 
completely dismiss a checklist must click on the “Completed” button 
at the bottom of each checklist. 

• For procedure steps that require another checklist, a hyperlink is 
provided. Hyperlink will be in blue; clicking on it will take you to 
hyperlinked section. 

• Schematics, graphics, and tables - can double click to enlarge, 
although this function has not been made available on all schematics, 
tables, and diagrams as of yet. 

Instructor: Ask to see if there are any questions up to this point. 

TREND MODE 

• Not applicable in this version. 

• When inserted will give the user a graphical view of system 
performance during aircraft run up and shutdown operations. 
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DATA MODE 


Functions 

• Corresponds to an electronic calculation of performance data 
information that was formerly done by hand. 

• Currently offers two types of electronic calculations — ability to 
maintain level flight (single engine) and range menu. 

Lavout/Navigation 

• Navigation here works similar to that within the checklist mode (in 
regards to most buttons). Operation of first two rows of blue buttons 
and operation of mid-page yellow buttons remain the same. 

• Within this mode there are two levels of yellow buttons: The first is 
the main menu (illustrate), while the second is a sub-menu (illustrate). 
Operate same as yellow buttons within checklist mode. 

Actual Performance Calculations 

• Once get into actual performance calculation screen (single engine 
level flight or range chart) you will see something that looks as 
follows: 

o On left will see boxes with PA, gross weight, and either OAT 
or fuel depending on calculation type. 

o To change the values currently in the boxes the user must use 
the keypad to the right. Simply click on the box on the left for 
which the value you want to change - the present value will 
disappear. Now click on the appropriate new value using the 
keypad and click on enter. Once you click enter the new 
outcome data will be calculated. 

o You will also notice a reset button on the keypad, clicking this 
will set the values back to their preset values. 

• The range chart will provide you with an electronic calculation of 
velocity, torque, and maximum range for a given set of values. 

o Within this chart also have a button that will let you indicate 
whether anti-ice is on or off and calculates values accordingly. 
Today we are going to leave it off. 

• The single engine level flight chart operates the same in terms of 
entering data values into the calculator, however in addition to the 
calculator there is a graphic chart. 

o Vmin and V max automatically calculated and displayed. 

o Chart buttons next to PA and Gross Weight indicates what x 
axis in chart will represent 

o When Gross Weight button is active (yellow) the chart on 
bottom represents the Vmin and Vmax values over a range of 
gross weights. The arrows indicated the Vmin and Vmax for 
the gross weight shown on the screen. 
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o When Pressure Altitude button is active (yellow) chart will 
show Vmin and Vmax across a range of pressure altitudes. To 
do so select chart button next to PA box. 


Final Information 

• Back/fwd, history, and bookmark navigational tools remain the same 
and are used the same. 

• Replacing the PCL navigational tool found within the checklist mode 
is the Calc navigational tool. Calc tool has two main functions: 

o Can take you back to the main menu for performance data 
(illustrate) 

o Click on close calc to close any open performance calculators. 
If more than one performance calculator is open at once, the 
currently active one will be yellow. Again, you can go back 
and forth between calculators by clicking on associated yellow 
tabs. 

Instructor: Ask if there are any questions up to this point. 

ALERT MODE 

• Not applicable in current version 

• Will be used to present conventional cautions, warnings, advisories, 
and eventually HUMS information to the aircrew. 

NATOPS MODE 


Function(s) 

• Provide access to NATOPS text in format similar to paper copy. 


Use of 

• The main menu of the NATOPS mode contains a table of contents that 
lists each of the individual chapters within the paper NATOPS. Find 
the chapter that contains the relevant information and click on the 
chapter to expand it into its individual sections. 

• Tree diagram is then expanded and section headings within the tree are 
preceded by a if this button is clicked that portion of the tree will 
collapse and a “+” will appear before the heading. Clicking on the “+” 
will expand the tree again if need be. Once can’t expand tree any 
more the actual text pertaining to selected section will appear (just as 
in paper NATOPS). 

• Note: Some of the major sections (e.g., section 3) contain general 
information in addition to more specific information contained under 
subsections (e.g., 3.1, 3.2). Means at this point you must be careful 
where you click to expand or might not get the information you 
intended - later sections with general information will be coded 
differently (color). 
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• This mode also contains hyperlinks (in blue - clicking on them will 
take you to related section), schematics, diagrams, and figures - no 
foldouts. 

Navigational tools 

• Navigational tools such as history, bookmark, back/forward remain the 
same within this mode. 

• Search tool is new. Clicking on the search function will allow three 
options: Other Docs, Index, or TOC (table of contents). 

- TOC: Takes you to a table of contents organized by individual 
chapters (also the main menu of the NATOPS mode). 

Index: Takes you to an alphabetical index. Click on letter to 
see topics under that letter. 

Other Docs: Contains AAW and SOP. (AAW option is still 
under construction.) 

• Hint: May be easiest to use TOC or Index within search tool to 
maneuver around this mode, as back/forward tool takes you back and 
forward to actual text, but not to the actual tree diagrams within this 
mode. 

Instructor: Ask if there are any questions up to this point. 

NOTES MODE 

• Electronic note pad 

• Probably not that useful within scenarios will be given today. 

• Allows users to take notes in electronic ink. Will use stylus. 

• Secondary buttons, clear, close, save, page (create new page - plain, 
AAW, or SOP (later two under construction) 

Instructor: At this point ask if have any questions about any of sections covered. If not 
explain that next they will be given some time to further familiarize themselves with the 
IE-NATOPS by practicing several general categories of items. Will have a couple more 
practice sessions after this one. 

Ill INTERACTIVE SCENARIO BASED PRACTICE (DO NOT READ) 

A. FREE TIME PRACTICE/FAMILARIZATION (Do not read) 


This next portion of training is going to allow you to have some hands-on practice with 
the EE-NATOPS. Basically, I want you to take about 10 minutes and familiarize 
yourself/complete the actions on this checklist. If you have any questions feel free to ask. 

Instructor: Hand out “Objectives During First Practice Session”. Basically the sheet 
requires them to cover the following: 

CHECKLIST MODE 
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• Navigating within and between checklists 

• Completion of a checklist 

• Use of back/forward button 

• Switching to the various types of checklists (EP, NP, Special procedures, etc...) 
NATOPSMODE 

• Navigating through the NATOPS chapters 

-make sure you get to the actual text at least once 

• Use of search function using toe 

• Use of search function using Index 
DATA MODE 

• Navigation to one of two performance charts 

• Change the values within this chart to see how electronic calculation works 
MISCELLANEOUS 

• Use of history button to get back to a prior section of your choice 

• Use of bookmark function (make a bookmark) 


Do you have any questions? 

B. PRACTICE SCENARIOS (Don’t Read) 

DURING THIS PRACTICE SESSION WE WILL BEGIN FOLLOWING SOME OF THE 
PROTOCOLS THAT YOU WILL BE REQUIRED TO FOLLOW DURING THE ACTUAL 
SCENARIOS SO THAT YOU WILL BEGIN TO GET USED TO THEM. AT THIS 
POINT, WHEN YOU GO TO EACH OF THE SECTIONS THAT I ASK YOU TO, 
PLEASE VERBALIZE THE PATH YOU ARE TAKING AND IF COMPLETING A 
CHECKLIST, INDICATE THAT EACH ITEM IS COMPLETE AS YOU READ IT. FOR 
EXAMPLE, CLICKING ON NP, CLICKING ON TAXI CHECKLIST, STEP 1 
COMPLETED, STEP 2 COMPLETED, CHECKLIST COMPLETE (MEANS YOU HAVE 
CLICKED ON THE “COMPLETE" BUTTON AT THE BOTTOM OF THE 
CHECKLIST). ITS VERY IMPORTANT FOR US THAT YOU VERBALIZE THE PATH 
YOU ARE TAKING. 

WE WILL DO A COUPLE OF SEARCHES WITHIN EACH OF THE MODES THAT 
YOU MIGHT BE REQUIRED TO USE WITHIN THE ACTUAL STUDY - STARTING 
WITH THE CHECKLIST MODE... 

SEARCHES WITHIN DATA MODE (DON’T READ) 

1) Within checklist mode, go to Normal Procedures 

• Find and open “Taxi Checklist” 

• Complete “Taxi Checklist” 

2) Within checklist mode, go to Emergency Procedures 

• Go to mission equipment malfunction 

• Go to Emergency Jettison Releases 
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• Go to Cargo Hook Emergency Release Checklist - find but do not 
complete 

3) Get back to Normal Procedures ( Evaluator: may use back or directly click on NP 
button) 

4) Use history button to get back to Emergency Procedures 

5) Is the PCL button currently active? What does this tell you? 

- Go in, complete the open checklist 

Now we will do a few searches within the data mode. 

SEARCHES WITHIN DATA MODE (DON’T READ) 

1) Within the data mode, go to Range/Level Flight Performance Menu 

• Click on Range Menu 

• Click on Maximum Range 

• Enter 3,000 PA - what does this make the velocity become? 

• Reset all functions... 

• Return to main menu for data mode (use calc function or back button) 

2) Within the data mode, go to Emergency Operation Menu 

• Click on Ability to Maintain Level Flight 

• Change x axis to reflect Gross Weight or PA ( whichever not currently the 
x axis) 

• Enter the following new values: PA=3,000 and WT=16,000 

• Read off the corresponding values calculated 

• Close all calculators 

During the actual study you will be timed in terms of how long it takes you to access 
various sections of the IE-NATOPS. As such, within the last couple of searches that we 
are going to do, you will notice that on the last item to be completed within each search I 
will ask you to let me know when you are done. This is so you get used to letting us 
know when you have reached the answer or piece of information you are looking for 
because within the actual study, once you have found the final answer and say you are 
done is when we will stop the clock. Remember to verbalize your path. 

Next, we will perform a few searches within NATOPS mode. 

Searches within NATOPS mode {don’t read) 

1) Within the NATOPS mode, use the Table of Contents (TOC) to go to “Systems” 
Chapter 

• Click on “Electrical Systems” 

• Click on “AC Electrical Systems” 

• Click on the hyperlink, and find information contained within. 

2) Return to main NATOPS menu {use either search function or back button) and let 
me know when you are there. 
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3) Within the NATOPS mode, use the index function to Main Landing Gear and 
information contained within 

• Scroll down to bottom of information so will see schematic/table, when 
you have reached the end let me know. 

4) Get back to main menu using whatever didn’t use before (Search function or back 
button) and let me know you are done. 

The next couple of searches will cut across several of the modes. 

Combination searches/Miscellaneous ( don’t read) 

1) Use history function to return to the Main Landing Gear section and let me know 
when you have found it. 

2) Go to the checklist mode, EP checklist, and find information pertaining to flight 
characteristics, particularly settling with power... 

• When trainee is halfway through memory items (#3) have them go to the 
data mode 

- Find the maximum range calculator 

- Enter PA=3,000 

- What are the new calculations? 

• Go back and complete the rest of the EP checklist ( can use back button, 
history button (maybe), or PCL button (easiest)) - let me know when you 
are done 

Do you have any questions? 

G.2 First Practice Session Objectives 

Objectives During First Practice Session: IE-NATOPS 

DURING THE NEXT 10 MINUTES PLEASE MAKE SURE YOU GET 
FAMILIAR WITH EACH OF THE FOLLOWING ACTIONS. IF YOU HAVE 
ANY QUESTIONS, PLEASE FEEL FREE TO ASK. 

CHECKLIST MODE 

_ Navigating within and between checklists 

_ Completion of a checklist 

_ Use of back/forward button 


Switching to the various types of checklists (EP, NP, Special procedures, 
etc...) 
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NATOPS MODE 


_ Navigating through the NATOPS chapters 

-make sure you get to the actual text at least once 

_ Use of search function using toe 

_ Use of search function using Index 

DATA MODE 

_ Navigation to one of two performance charts 

_ Change the values within this chart to see how electronic calculation 

works 

MISCELLANEOUS 

_ Use of history button to get back to a prior section of your choice 

_ Use of bookmark function (make a bookmark) 
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Appendix H. Experiment 1 — Experimenter-Directed 
Searching 

H. 1 Final Timed Practice Session - IE-NATOPS 


**Evaluator: In addition to participant saying stop when search is complete you also 

should say stop because the timer will be primarily looking to you. 

THIS IS THE FINAL PRACTICE SESSION AND DURING THIS SESSION WE WILL 

TIME YOUR SEARCHES TO GET A TYPE OF BASELINE AND ENSURE THAT YOU 

ARE COMFORTABLE WITH USING THE IE-NATOPS. AS WELL AS WE WILL BE 

TIMING YOUR SEARCHES DURING THE ACTUAL STUDY THIS WILL BEGIN TO 

GET YOU USED TO THE PROCEDURE. PLEASE REMEMBER TO: 

• VERBALIZE THE PATH YOU ARE TAKING - INCLUDING AS YOU 
COMPLETE EACH STEP OF THE CHECKLIST AND WHEN YOU 
HAVE COMPLETED THE ENTIRE THING (HIT COMPLETE 
BUTTON). IT'S VERY IMPORTANT THAT YOU GET IN THE HABIT OF 
VERBALIZING YOUR PATH BECAUSE DURING THE ACTUAL STUDY 
THERE WILL BE TIMES WHEN WE WILL HAVE TO PROVIDE YOU 
WITH UPDATES - THESE ARE CUED BY WHEN YOU REACH A 
CERTAIN POINT (HENCE THE VERBALIZATION). 

• AGAIN, YOU WILL NOTICE THAT ON THE LAST ITEM TO BE 
COMPLETED WITHIN EACH SEARCH I WILL ASK YOU TO LET ME 
KNOW WHEN YOU ARE DONE. THIS IS SO YOU GET USED TO 
LETTING US KNOW WHEN YOU HAVE REACHED THE ANSWER OR 
PIECE OF INFORMATION YOU ARE LOOKING FOR BECAUSE 
WITHIN THE ACTUAL STUDY, ONCE YOU HAVE FOUND THE FINAL 
ANSWER AND SAY YOU ARE DONE IS WHEN WE WILL STOP THE 
CLOCK. 

Search #1 

• Within the checklist mode, go to Normal Procedures 

o Go to “Start Checklist” and click on 
o Expand “Start Checklist” 

• Go to “Systems Check” and complete all relevant checklists, including “Cargo Hook 
Operational” checklist. (Note to evaluator: within this first checklist there is a 
hyperlink that takes them to a second checklist, should complete both checklists). 

• Return to the main menu for Normal Procedures and let me know once there (STOP 
CLOCK) 

Search #2 

• Use the History function to find “Flight Characteristics” 

• Return to Emergency Procedures main menu and let me know once there (STOP 
CLOCK) 
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Search #3 

• Within the checklist mode, go to Emergency Procedures 

o Go to “Sonar Malfunction” 

■ Go to “Sonar” 

■ Go to “Sonobuoy lithium battery venting” checklist, including the link 
to “Smoke and Fumes Elimination. ” {Note to evaluator: This checklist 
has a hyperlink to checklist entitled “Smoke & Fumes Elimination. ”) 

■ Complete all relevant checklists 

• Within checklist mode, go to Reference Data Section 

o Go to “Engine Limits” 

■ Click on “Engine Starter Limits” and once within the checklist scroll 
to the bottom and complete 

o Make sure all checklists are completed and closed 

• Return to Emergency Procedures main menu and let me know (STOP CLOCK) 

• How can you double check to make sure all checklists have been completed/are 
closed? 

Search #4 

• Go to NATOPS mode 

o Go to the “Avionics” section 

■ Open “Mission Avionics” 

• Use Search function to find “Aircrewman Only Brief’ (Evaluator: Easiest to use 
index ) 

o Scroll to the bottom of “Aircrewman Only Brief’ and let me know when there 
(STOP CLOCK) 

Search #5 

• Go to the main menu for the NATOPS mode (TOC) (Evaluator: Easiest to use 
Search -TOC) 

• Go to “Aircraft Operating Limitations” 

o Go to Weight Limitations 

o Go to Weight Limits (Evaluator: As soon as trainee gets here go to next step) 

• Go to main menu for Data Mode 

o Go to emergency operations menu 

■ Go to ability to maintain level flight, single engine 

• Change the Wt to 17,000 and make Wt the “x” axis 

• What are the new values? 
o Close the calculator 

• Return to main menu of NATOPS mode (TOC) and let me know when you are there 
(STOP CLOCK) 

Search #6 

• Within checklist mode, go to “Emergency Procedures” 

• Go to section pertaining to “Warning, caution, advisory light” 
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o Find Advisory Light section and click on 

■ Find “Search Light On” section and click on ( Evaluator: Once here 
proceed) 

• Go to NATOPS mode to find further information on Search Light 

o Evaluator: Easiest way is to use Index function, again once found move on 

• Go to main menu for Data Mode 

o Find “Range Level Flight” 

■ Click on “Range” 

• Click on “Max Range” 

o Set Gross Weight = 17,000 
o What are the new values 

• Close calculator 

• Use History Function to get back to Main Menu for Data Mode and let me know 
when you are there ( STOP CLOCK) 


Ask if there are any final questions.... If not this concludes our training session. 


H.2 Final Timed Practice Session - Paper NATOPS 

**Evaluator: In addition to participant saying stop when search is complete you also 
should say stop because the timer will be primarily looking to you. 

Before we begin the actual study in which you use the paper NATOPS to find certain 
information we will do a couple of practice searches to get a type of baseline and give 
you some practice on the procedures that we will be using. 

During these practice searches, we will begin following some of the protocols that you 
will be required to follow during the actual scenarios so that you will begin to get used to 
them. At this point, when you go to each of the sections that I ask you to, please 
verbalize the path you are taking and if completing a checklist, indicate that each item is 
complete as you read it. For example, clicking on NP, clicking on taxi checklist, step 1 
completed, step 2 completed, checklist complete (means you have clicked on the 
“complete” button at the bottom of the checklist). It’s very important for us that you 
verbalize the path you are taking. 

During the actual study you will be timed in terms of how long it takes you to access 
various sections of the NATOPS. As such, within the last couple of searches that we are 
going to do, you will notice that on the last item to be completed within each search I will 
ask you to let me know when you are done. This is so you get used to letting us know 
when you have reached the answer or piece of information you are looking for because 
within the actual study, once you have found the final answer and say you are done is 
when we will stop the clock. Remember to verbalize your path. 

Search #1 
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• In the PCL go to Normal Procedures section/tab 

o Find the Start Checklist 
o Complete the “Systems Check” checklist 

■ Including Cargo Hook Operational Checklist 

• {Note to evaluator: within the “Systems Check” checklist there is a hyperlink that 
takes them to a second checklist “Cargo Hook Operational”, should complete both 
checklists). 

• Close the PCL once you have completed both checklists. 

Search #2 

• In NATOPS go to the chapter on Emergency Procedures 

o Find section on “Flight Characteristics” and let me know when you have 
found this section 

• Close NATOPS {STOP CLOCK) 

Search #3 

• In the PCL go to the Emergency Procedures section 

o Find “Sonar Malfunction” 

■ Find section on “Sonar” 

■ Find and complete “Sonobuoy lithium battery venting” checklist, 
including the link to “Smoke and Fumes Elimination.” {Note to 
evaluator: This checklist has a hyperlink to checklist entitled “Smoke 
& Fumes Elimination. ”) 

■ Complete all relevant checklists 

• Now go to the Reference Data section within the PCL. 

o Find “Engine Limits” 

■ Find and complete “Engine Starter Limits” checklist 

• Return to the beginning of the Emergency Procedures section and let me know when 
you are there {STOP CLOCK) 

Search #4 

• Within NATOPS find the Avionics Chapter 

o Find the “Avionics” section 

■ Find “Mission Avionics” 

• Find section pertaining to “Aircrewman Only Brief’ {Evaluator: Easiest to use index) 

o Find the last step in “Aircrewman Only Brief’ and then close NATOPS 
{STOP CLOCK) 

Search #5 

• Using the menu in NATOPS find “Aircraft Operating Limitations” 

o Find section pertaining to “Weight Limitations” 

o Find section pertaining to “Weight Limits” {Evaluator: As soon as trainee 
gets here go to next step) 

• Find performance charts within NATOPS 
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o Find “Emergency Operations” 

■ Find “Ability to Maintain Level Flight, Single Engine” chart 

• If your GW is 17,000 

• PA=2,000 

• OAT=20 degrees Celsius 

• Torque=109% 

• What are Vmin and Vmax values? 

o When finished calculating close NATOPS. ( STOP CLOCK) 

Search #6 

• In PCL find “Emergency Procedures” section 

• Go to section pertaining to “Warning, caution, advisory light” 

o Find Advisory Light section 

■ Find “Search Light On” section (Evaluator: Once here proceed) 

• Go to NATOPS to find further information on Search Light 

o Evaluator: Easiest way is to use Index function, again once found move on 

• Find performance chart associated with Range of Level Flight in NATOPS 

■ Maximum Range Chart 

• Set Gross Weight = 17,000 

• PA=3000 

• Fuel=1090 

• Torque? No, but if yes 109% 

• What are the new values 

• Close NATOPS when done (STOP CLOCK) 


Ask if there are any final questions.... If not this concludes our 
training session. 
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Appendix I. Experiment 2 — Self-Directed Searching 

1.1 Instructions 


INFORMATION TO BE READ BEFORE BEGIN ACTUAL STUDY 

Now we are going to begin the actual study. The procedure for each scenario is twofold. 
First, we will hand you a scenario and after reading through it we will ask you to walk us 
through what you would do to solve the problem/questions asked if you were actually in 
the cockpit flying. For example, within the actual aircraft you may not break out the 
NATOPS very often or may pull out the PCL once the immediate danger has passed. 
After you have told us how you would handle solving the problem if within the actual 
aircraft, we will ask you to go through and actually show us the procedure(s) that you 
would use to look up the required information in the NATOPS, IE-NATOPS, or PCL. I 
imagine the best way to look at this second part is as if you were participating in an open- 
book annual NATOPS check. So within the second part of this procedure things may 
seem somewhat artificial. During the portion of the procedure where we are requiring 
you to actually find the information within the PCL, NATOPS, etc. The protocols that 
you will be asked to follow are similar to those during your training: 

- As you will be timed, tell us when you are ready to start. 

- Again verbalize your path and as you are completing checklists read them out. 

- Let us know when you have completed your search and found the final piece of 
information that will allow you to answer all the relevant questions or actions 
requested - at this time we will stop the clock. 


Within the scenarios you should assume unless we tell you otherwise that the information 
given within the scenario is all the information you currently have access to. From time 
to time we may provide you with verbal information/updates as you go through some of 
the required checklists - this is why it is so important that you verbalize the path you are 
taking and the steps that you are doing in completing the checklists. 

1.2 IE-NATOPS Evaluation Scenarios 

Scenarios 


1. Current Aircraft Profile: Aircraft: SH-60F 

Heading: 325 

Altitude: 1500 ft 

Airspeed: -128 KIAS 

Available Fuel: 1690 lbs. 

Current Fuel Consumption Rate: 1088 lbs./hr 

Current Position from Destination: 80 NM SE 

Aircraft Gross Weight: 15500 lbs. 


Narrative: 
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You’re already in flight on a VFR cross-country training flight from airport A to 
airport B. You are currently 80 NM SE from your destination (as shown above). You’re 
scheduled to attend an air show at Airport B the following day. You should be able to 
make it to your destination before it goes IMC, however, you decide to look for an 
alternate just in case. The only nearby airport (airport C) that can provide the required 
fuel is 56 NM farther from your current position than airport B. You call in for a weather 
update for your destination. The controller reports that the cold front is moving in faster 
then forecasted, and that airport B will be going IMC for an unspecified amount of time 
prior to your arrival. Airport C will remain VFR for the duration of the flight; however, 
you will go IMC above 3000 ft. In addition, anti-ice is off. Can you make it to airport C 
without going below the minimum planned fuel requirement, or will you need to re-file 
an EFR flight plan for airport B? Let me know when you have found the answer. 


2. Current Aircraft Profile: 


Aircraft: 
Heading: 
Altitude: 
Airspeed: 
Available Fuel: 


Aircraft Gross Weight: 


SH-60F 

275 

2000 ft 
-127 KIAS 
800 lbs. 
1100 lbs./hr 
20 NM SE 

16500 lbs. 


Current Fuel Consumption Rate: 
Current Position from Destination: 


Narrative: 


You’re approaching the first refueling stop on a cross-country training flight. An 
unexpected strong head wind has delayed your arrival and increased your fuel 
consumption so you need to land as soon as possible before going below the minimum 
planned fuel requirement. The weather is beginning to close in from the north so you 
contact the controller for a weather update. The controller informs you that the airfield 
is going IMC. Looking at the sectional you see an airport 10 NM south of your current 
position. You contact that airfield for weather and available services. The controller says 
the airfield is VFR and that the only fuel they provide is JP-4/ JET B. Can this fuel be 
used, and, if so, what restrictions/considerations come with it? Let me know when you 
have found the answers to both questions 


3. Current Aircraft Profile: Heading: 

275 

Altitude: 

1500 ft 

Airspeed: 

-128 KIAS 

Available Fuel: 

1690 lbs. 

Current Fuel Consumption Rate: 

1088 lbs./hr 

Aircraft Gross Weight: 

15500 lbs. 


Narrative: 
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You’ve taken off from the ship to conduct SAR training. One hour into the 
mission you detect smoke in the cockpit from an unknown electrical source. The flight 
back to the nearest ship will take approximately 20 minutes. You immediately head to 
the nearest ship. Perform the appropriate procedures and isolate the source of the smoke. 
Let me know when you have isolated the source of the smoke and performed appropriate 
procedures. 


4. Current Aircraft Profile: Heading: 275 

Altitude: 3000 ft 

Airspeed: -128KIAS 

Available Fuel: 4000 lbs. 

Current Fuel Consumption Rate: 1050 lbs./hr 

Aircraft Gross Weight: 16500 lbs. 


Narrative: 

You’ve just completed some SAR training and are returning to the ship. You’re 
about 30 minutes from the ship when the Ng begins to drop. It goes below 55% and the 
#1 ENG OUT caution light illuminates. Ng goes to zero, however, Nr and all other 
indicators for the #1 and #2 engines are reading normal. No vibrations are indicated. 
Can the alternator be the cause of the problem? (Find reference in NATOPS). Are there 
any procedures associated with this problem? (Refer to NATOPS as required). Let me 
know when you have found the answers to both questions. 


5. Current Aircraft Profile: Aircraft SH-60F 

Heading: 275 

Altitude: On the Ground 

Airspeed: 0 KIAS 

Available Fuel: 4000 lbs. 

Current Fuel Consumption Rate: 0 lbs./hr 

Aircraft Gross Weight: 16500 lbs. 

OAT: 20 ° F 


Narrative: 

It is 0600, and you are going on a training flight. It is the first flight of the day for 
this aircraft. You’ve already completed the pre-start and systems checks and are about to 
begin starting engines checklist. Find this checklist and read through procedures as if 
you where in the cockpit starting the #1 engine. Evaluator will provide any anomalous 
systems indications that may arise, otherwise assume all indications are reading normal 
during systems check. Complete the checklist for engine #1 before answering the 
following question. Are environmental factors the cause for the possible anomalous 
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indication? Find the NATOPS reference that discusses this indication and let me know 
when you have found it. 


6. Current Aircraft Profile: Aircraft: SH-60F 

Heading: 360 

Altitude: 1500 ft 

Airspeed: -124KIAS 

Available Fuel: 2000 lbs. 

Current Fuel Consumption Rate: 1080 lbs./hr 

Aircraft Gross Weight: 16500 lbs. 

OAT: 45 ° F 


Narrative: 

After an hour of SAR training you begin to experience low-frequency lateral 
vibration in the aircraft. All systems indicators are reading normal. 30 minutes later the 
vibrations are beginning to increase. You decide to return to ship. As you turn the 
aircraft you notice increased vibration, slight fluctuations in your Np/Nr, and slight pilot 
induced oscillations. All other systems indicators are reading normal. What is the 
suspected main rotor malfunction? Find information on malfunction in NATOPS. Go 
through emergency procedures as required. Let me know when you have completed this 
scenario. 


7. Current Aircraft Profile: Aircraft: SH-60F 

Heading: 090 

Altitude: 1500 ft 

Airspeed: -124KIAS 

Available Fuel: 1600 lbs. 

Current Fuel Consumption Rate: 1080 lbs./hr 

Aircraft Gross Weight: 16500 lbs. 

OAT: 10 °C 


Narrative: 

You’ve completed the day’s mission and are returning to the ship. You’re about 
20 minutes out when you hear very muffled explosion like sounds in the rear. You 
immediately check your systems instruments for anomalous indications. You detect a 
rapid increase in your #1 TGT and decrease in #1 Ng. What is the problem? Locate the 
appropriate procedures and perform. Once the situation has been stabilized, maintain 
level flight at 1500 ft for remainder of flight back to ship. What are your Vmin and 
Vmax readings at this point? (Use Single Engine Level Flight (SELF) Calculator if using 
the PC or the chart if using the NATOPS manual.) 
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8. Narrative: 


You are on a return flight back to your home base when the Communication 
System Controller (CSC) fails. The automatic CSC failure backup mode becomes 
enabled. Which internal and external communications functions become available in this 
mode for all crew stations? Are these secure communications? Do all crew stations have 
external communication capabilities? Use NATOPS to obtain information. Using the 
Communication System Block Diagram, identify the systems that interact with the 
Communication System Control (CSC) by pointing to them in the diagram. Let me know 
when you feel you have all the answers to the questions posed in this scenario. 


9. Current Aircraft Profile: Aircraft: SH-60F 

Heading: 020 

Altitude: 1500 ft 

Airspeed: -124KIAS 

Available Fuel: 4000 lbs. 

Current Fuel Consumption Rate: 1080 lbs./hr 

Aircraft Gross Weight: 16500 lbs. 


Narrative: 

You’ve taken off from your ship and are about 15 minutes into the flight when the 
controller instructs you to turn left to 270 and climb to 3000 ft. You immediately place 
the aircraft into a climbing left turn, when you suddenly get #1 and #2 CONV, AC ESS 
BUS OFF, AFCS DEGRADED, and STABILATOR caution lights illuminated. Identify 
the emergency and perform the appropriate procedures. Will the stabilator automatic 
mode be enabled in this condition? Let me know when you have answered this question 
and performed the actions required by the scenario. 


10. Narrative: 

You’ve just completed the first leg of a combined cross-country training/cargo 
transport flight, and have just landed at a small airport for re-fueling. Your aircraft is 
parked on a soft grassy field with a heading of 180. With the cargo and a full load of fuel 
the aircraft weighs approximately 19,000 lbs. Winds are 270/15 gusting to 20. With all 
this in mind you slowly pick the aircraft up to a hover. Suddenly the nose of the aircraft 
begins to yaw right. You apply more left pedal, but the yaw continues. What is 
happening to the aircraft and what are the factors that can contribute to this situation? 
Find the information in NATOPS. Let me know when you have found the answer to 
these questions. 
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Appendix J. Summary Statistics 


Table J-l. Summary Statistics for Key Study Variables 
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Table J-2. Correlations Between Demographic Variables and Key Study Variables 

Rank Age Months in Time in # of Time in Time in Aircraft Approx. Approx. Approx. 
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Appendix K. Ancillary Questionnaire Analyses 

Additional analyses were performed to address potential differences on the survey 
questions as a function of specific grouping variables obtained from the demographic 
data. The following category variables were examined: 

1) rank—LT vs. LCDR, 

2) age—less than or equal to 30 years vs. greater than 30 years, 

3) time in service—less than or equal to 105 months vs greater than 105 
months, 

4) total hours in the H-60F—less than or equal to 600 hours vs. greater than 
600 hours, 

5) approximate time in the H-60F in the last 90 days—less than or equal to 
15 hours vs. greater than 15 hours, 

6) approximate time in the H-60F simulator—less than or equal to 100 hours 
vs. greater than 100 hours, 

7) total time in rotary wing—less than or equal to 1500 hours vs. greater 
than 1500 hours, and 

8) total flight time—less than or equal to 1600 hours vs. greater than 1600 
hours. 

Note that with the exception of rank (there was little variability here — participants were 
either LT or LCDR), the criteria for determining the two groupings for each category was 
based on median values for each category. That is, in most cases, 50% of the participants 
were in each grouping (e.g., for the age category, half of the participants were less than 
30 years, while the other half were greater than 30 years). Traditional t-tests for 
independent groups were conducted on each of these categories for the particular groups 
that were developed. Each of these different categories will now be examined separately. 

K.l Rank 


Table K-l gives the means for each survey question as a function of rank—LT vs. 

LCDR. None of the t-tests were significant, indicating no difference in response to the 
post-experiment questions as a function of military rank. 

While there were no significant differences noted, several interesting patterns seemed to 
emerge here when looking at rank. First, the difference in ratings between NATOPS and 
IE-NATOPS appear to be greater for the more senior personnel. That is, when looking at 
6b, for example, it seems that the senior rank pilots rate IE-NATOPS higher on this 
dimension (ease of interface manipulation) than the more junior officers (1.33 for seniors 
vs. 1.57 for juniors). This same pattern seems evident for 7b (accessing data), 9b (finding 
data to resolve the scenarios), and on 10b (searching multiple sections). Juniors rated IE- 
NATOPS more favorably only on 8b (trouble in reading and understanding NATOPS 
data). Responses on 1 la-f seems mixed between senior and junior personnel, while 
responses on 12a and 12b seem to show greater agreement on the usefulness of these 
functions (bookmark and performance charts) on the part of juniors compared to that of 
more senior officers. 
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Table K-l. Mean Response Rates as a Function of Rank 



2. Amount of 

1 = too little 

2.00 

2.33 

IE-NATOPS training 

2 = sufficient 

3 = too much 




3. Amount of 

IE-NATOPS practice 

1 = too little 

2 = sufficient 

3 = too much 

2.00 

2.33 

4. Difficulty of IE-NATOPS training 
scenarios 

1 = too easy 

2 = sufficient 

3 = too complex 

2.00 

2.33 

5. Confidence for in-flight use of IE- 
NATOPS 

1 = very confident 

5 = very uncertain 

1.71 

1.33 

6a. Paper NATOPS interface easy to 
manipulate 

1 = strongly agree 

5 = strongly disagree 

2.71 

3.00 

6b. IE-NATOPS interface easy to 
manipulate 

1 = strongly agree 

5 = strongly disagree 

1.57 

1.33 

7a. Always knew how to use paper 
NATOPS interface 

1 = strongly agree 

5 = strongly disagree 

2.00 

2.67 

7b. Always knew how to use IE-NATOPS 
interface 

1 = strongly agree 

5 = strongly disagree 

2.00 

1.67 

8a. Had no trouble reading & 
understanding paper NATOPS 

1 = strongly agree 

5 = strongly disagree 

1.29 

1.33 

8b. Had no trouble reading & 
understanding IE-NATOPS 

1 = strongly agree 

5 = strongly disagree 

1.43 

2.00 


9a. Had no trouble finding data with paper 
NATOPS 

1 = strongly agree 

5 = strongly disagree 

2.43 

2.67 

9b. Had no trouble finding data with paper 
NATOPS 

1 = strongly agree 

5 = strongly disagree 

2.14 

2.00 

10a. Easy to search multiple sections with 
paper NATOPS 

1 = strongly agree 

5 = strongly disagree 

2.57 

1.33 

10b. Easy to search multiple sections with 
IE-NATOPS 

1 = strongly agree 

5 = strongly disagree 

1.57 

1.33 

11. For IE-NATOPS: 




11 a. History function easy to use 

1 = strongly agree 

5 = strongly disagree 

2.14 

2.33 

11 b. Bookmark function easy to use 

1 = strongly agree 

5 = strongly disagree 

2.14 

2.00 

1 lc. Navigation Keys easy to use 

1 = strongly agree 

5 = strongly disagree 

1.71 

1.67 

1 Id. Tab Feature in Checklists easy to use 

1 = strongly agree 

1.43 

1.33 
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5 = strongly disagree 



1 le. Search Feature in NATOPS easy to 
use 

1 = strongly agree 

5 = strongly disagree 

1.43 

2.33 

1 If. Performance Charts easy to use 

1 = strongly agree 

5 = strongly disagree 

1.43 

1.67 

12. For IE-NATOPS: 




12a. Bookmark Function would be 
operationally useful 

1 = strongly agree 

5 = strongly disagree 

2.29 

3.00 

12b. Performance Charts would be 
operationally useful 

1 = strongly agree 

5 = strongly disagree 

1.57 

2.00 


K.2 Age 

Table K-2 gives the means for the survey questions as a function of age—less than 30 
years vs. greater than 30 years. None of these comparisons were significant, although it 
would appear that the younger group had greater agreement on questions 6b, 7b, 8b, 9b, 
but not 10b. These questions specifically address IE-NATOPS dimensions. The greatest 
difference in groups comes with 8b (M young = 1.20 vs. M older = 2.00). The younger 
group was more in agreement than the older group (p = .06) with the statement that they 
had no trouble reading or understanding the NATOPS data. This latter finding parallels 
what was found for junior rank pilots in the analysis of rank. This should not be 
surprising, as there was a strong correlation between rank and age (r = .76). 


Table K-2. Mean Response Rates as a Function of Age 


Item 

Scoring Scale 

<30 

yrs. 


1. IE-NATOPS training organization 

1 = too loose 

2 = perfect 

3 = too regimented 

2.00 

2.20 

2. Amount of 

IE-NATOPS training 

1 = too little 

2 = sufficient 

3 = too much 

2.00 

2.20 

3. Amount of 

IE-NATOPS practice 

1 = too little 

2 = sufficient 

3 = too much 

2.00 

2.20 

4. Difficulty of DE-NATOPS training 
scenarios 

1 = too easy 

2 = sufficient 

3 = too complex 

2.00 

2.20 

5. Confidence for in-flight use of IE- 
NATOPS 

1 = very confident 

5 = very uncertain 

1.60 

1.60 

6a. Paper NATOPS interface easy to 
manipulate 

1 = strongly agree 

5 = strongly disagree 

2.60 

3.00 

6b. DE-NATOPS interface easy to 
manipulate 

1 = strongly agree 

5 = strongly disagree 

1.40 

1.60 

7a. Always knew how to use paper 

1 = strongly agree 

2.20 

2.20 
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NATOPS interface _ 

7b. Always knew how to use IE-NATOPS 

interface _ 

8a. Had no trouble reading & 

understanding paper NATOPS _ 

8b. Had no trouble reading & 

understanding IE-NATOPS _ 

9a. Had no trouble finding data with paper 

NATOPS _ 

9b. Had no trouble finding data with paper 

NATOPS _ 

10a. Easy to search multiple sections with 

paper NATOPS _ 

10b. Easy to search multiple sections with 

IE-NATOPS _ 

11. For IE-NATOPS: _ 

11 a. History function easy to use 


11 b. Bookmark function easy to use 


1 lc. Navigation Keys easy to use 


1 Id. Tab Feature in Checklists easy to use 


I le. Search Feature in NATOPS easy to 

use _ 

II f. Performance Charts easy to use 


12. For IE-NATOPS: _ 

12a. Bookmark Function would be 

operationally useful _ 

12b. Performance Charts would be 
operationally useful _ 


5 = strongly disagree 
1 = strongly agree 
5 = strongly disagree 
1 = strongly agree 
5 = strongly disagree 
1 = strongly agree 
5 = strongly disagree 
1 = strongly agree 
5 = strongly disagree 
1 = strongly agree 
5 = strongly disagree 
1 = strongly agree 
5 = strongly disagree 
1 = strongly agree 
5 = strongly disagree 


1 = strongly agree 
5 = strongly disagree 
1 = strongly agree 
5 = strongly disagree 
1 = strongly agree 
5 = strongly disagree 
1 = strongly agree 
5 = strongly disagree 
1 = strongly agree 
5 = strongly disagree 
1 = strongly agree 
5 = strongly disagree 


1 = strongly agree 
5 = strongly disagree 
1 = strongly agree 
5 = strongly disagree 


1.80 


1.20 


1.20 


2.60 


2.00 


2.60 


1.60 


2.00 


1.40 


2.00 


2.40 


2.20 


1.80 


1.40 


2.20 


2.20 


1.40 


1.40 


1.40 


1.40 


2.20 


2.00 


2.00 


1.40 


2.00 


1.60 


2.40 2.50 


1.60 1.75 


K.3 Time in Service 


Table K-3 presents the mean response rates as a function of time in service — less than 
105 months represented the less experienced group, while greater than 105 months 
represented the more experienced group. None of the survey question comparisons were 
significant for this category. Again, like in the previous category of age, less experienced 
pilots agreed more with IE-NATOPS items (6b, 7b, 8b, 9b) than the more experienced 
group. Only item 10b was judged higher in agreement for the more experienced group in 
comparison to the less experienced group. 

That these results are similar to the results reported for age is understandable in that the 
correlation between age and time in service is quite high (r = .93). It is not clear why we 
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do not see the same pattern for rank, since rank is significantly correlated with both age (r 
= .76) and time in service (r = .88). 


Table K-3. Mean Response Rates as a Function of Time in Service 


Item 

Scoring Scale 

<105 

mos. 

>105 

mos. 

1. IE-NATOPS training organization 

1 = too loose 

2 = perfect 

3 = too regimented 

2.00 

2.25 

2. Amount of 

IE-NATOPS training 

1 = too little 

2 = sufficient 

3 = too much 

2.00 

2.25 

3. Amount of 

IE-NATOPS practice 

1 = too little 

2 = sufficient 

3 = too much 

2.00 

2.25 

4. Difficulty of IE-NATOPS training 
scenarios 

1 = too easy 

2 = sufficient 

3 = too complex 

2.00 

2.00 

5. Confidence for in-flight use of IE- 
NATOPS 

1 = very confident 

5 = very uncertain 

1.60 

1.50 

6a. Paper NATOPS interface easy to 
manipulate 

1 = strongly agree 

5 = strongly disagree 

2.60 

3.50 

6b. IE-NATOPS interface easy to 
manipulate 

1 = strongly agree 

5 = strongly disagree 

1.40 

1.50 

7a. Always knew how to use paper 
NATOPS interface 

1 = strongly agree 

5 = strongly disagree 

2.20 

2.50 

7b. Always knew how to use IE-NATOPS 
interface 

1 = strongly agree 

5 = strongly disagree 

1.80 

2.00 

8a. Had no trouble reading & 
understanding paper NATOPS 

1 = strongly agree 

5 = strongly disagree 

1.20 

1.50 

8b. Had no trouble reading & 
understanding EE-NATOPS 

1 = strongly agree 

5 = strongly disagree 

1.20 

1.75 

9a. Had no trouble finding data with paper 
NATOPS 

1 = strongly agree 

5 = strongly disagree 

2.60 

2.75 

9b. Had no trouble finding data with paper 
NATOPS 

1 = strongly agree 

5 = strongly disagree 

2.00 

2.25 

10a. Easy to search multiple sections with 
paper NATOPS 

1 = strongly agree 

5 = strongly disagree 

2.60 

2.00 

10b. Easy to search multiple sections with 
EE-NATOPS 

1 = strongly agree 

5 = strongly disagree 

1.60 

1.25 

11. For IE-NATOPS: 




1 la. History function easy to use 

1 = strongly agree 

5 = strongly disagree 

2.20 

2.25 

1 lb. Bookmark function easy to use 

1 = strongly agree 

5 = strongly disagree 

2.20 

2.00 


90 

















1 lc. Navigation Keys easy to use 

1 = strongly agree 

5 = strongly disagree 

1.40 


lid. Tab Feature in Checklists easy to use 

1 = strongly agree 

5 = strongly disagree 

1.40 


1 le. Search Feature in NATOPS easy to 
use 

1 = strongly agree 

5 = strongly disagree 

1.40 

2.00 

Ilf. Performance Charts easy to use 

1 = strongly agree 

5 = strongly disagree 

1.40 

1.50 

12. For IE-NATOPS: 




12a. Bookmark Function would be 
operationally useful 

1 = strongly agree 

5 = strongly disagree 

2.40 

2.67 

12b. Performance Charts would be 
operationally useful 

1 = strongly agree 

5 = strongly disagree 

1.60 

1.67 


K.4 H-60F Flight Time 

Table K-4 gives mean response rates for the survey questions as a function of H-60F 
flight time — less than 600 hours vs. greater than 600 hours. Three statistically 
significant comparisons were noted; namely, 8a [t(8) = 2.45, p = .04], 1 lb [t(8) = 2.89, p 
= .02], and 12a [t(8) = 2.97, p = .02]. In addition, several survey questions showed levels 
approaching significance; 7b (p = .09) and 10b (p = .07). The significant difference on 8a 
indicates that the pilots with more H-60F experience (flight time) agreed more strongly 
with the statement that they had no trouble in reading and understanding the NATOPS 
data using traditional NATOPS as a source. The significant difference on 1 lb indicated 
that pilots with more experience were not as likely to agree that the bookmark function 
was easy to use. Finally, the difference between groups on 12a supported 1 lb in that 
more experienced H-60F pilots did not find the bookmark function as useful as less 
experienced H-60F pilots in the operational environment. Considering 7b, while not 
significant, does show that more experienced H-60F pilots agreed more with the 
statement that they knew what to do to the interface to access data when using IE- 
NATOPS; evidently, less experienced pilots were not as comfortable with this aspect. 

The last comparison, dealing with 10b, indicates that less experienced pilots were more in 
agreement than experienced pilots that it was easy to access information when it came to 
searching for multiple sections. 


Table K-4. Mean Response Rates as a Function of H-60F Flight Time 


Item 

Scoring Scale 



1. IE-NATOPS training organization 

1 = too loose 

2 = perfect 

3 = too regimented 

2.00 

2.20 

2. Amount of 

IE-NATOPS training 

1 = too little 

2 = sufficient 

3 = too much 

2.00 

2.20 

3. Amount of 

IE-NATOPS practice 

1 = too little 

2 = sufficient 

2.00 

2.20 
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3 = too much 



4. Difficulty of IE-NATOPS training 
scenarios 

1 = too easy 

2 = sufficient 

3 = too complex 

2.00 

2.20 

5. Confidence for in-flight use of IE- 
NATOPS 

1 = very confident 

5 = very uncertain 

1.40 

1.80 

6a. Paper NATOPS interface easy to 
manipulate 

1 = strongly agree 

5 = strongly disagree 

3.20 

2.40 

6b. IE-NATOPS interface easy to 
manipulate 

1 = strongly agree 

5 = strongly disagree 

1.60 

1.40 

7a. Always knew how to use paper 
NATOPS interface 

1 = strongly agree 

5 = strongly disagree 

2.40 

2.00 

7b. Always knew how to use IE-NATOPS 
interface 

1 = strongly agree 

5 = strongly disagree 

2.20 

1.60 

8a. Had no trouble reading & 
understanding paper NATOPS * 

1 = strongly agree 

5 = strongly disagree 

1.60 

1.00 

8b. Had no trouble reading & 
understanding IE-NATOPS 

1 = strongly agree 

5 = strongly disagree 

1.80 

1.40 

9a. Had no trouble finding data with paper 
NATOPS 

1 = strongly agree 

5 = strongly disagree 

2.60 

2.40 

9b. Had no trouble finding data with paper 
NATOPS 

1 = strongly agree 

5 = strongly disagree 

2.20 

2.00 

10a. Easy to search multiple sections with 
paper NATOPS 

1 = strongly agree 

5 = strongly disagree 


2.00 

10b. Easy to search multiple sections with 
IE-NATOPS 

1 = strongly agree 

5 = strongly disagree 

1.20 

1.80 

11. For IE-NATOPS: 




11 a. History function easy to use 

1 = strongly agree 

5 = strongly disagree 

2.00 

2.40 

1 lb. Bookmark function easy to use * 

1 = strongly agree 

5 = strongly disagree 

1.60 

2.60 

1 lc. Navigation Keys easy to use 

1 = strongly agree 

5 = strongly disagree 

1.80 

1.60 

1 Id. Tab Feature in Checklists easy to use 

1 = strongly agree 

5 = strongly disagree 

1.20 

1.60 

1 le. Search Feature in NATOPS easy to 
use 

1 = strongly agree 

5 = strongly disagree 

1.80 

1.60 

1 If. Performance Charts easy to use 

1 = strongly agree 

5 = strongly disagree 

1.40 

1.60 

12. For IE-NATOPS: 




12a. Bookmark Function would be 
operationally useful * 

1 = strongly agree 

5 = strongly disagree 

1.75 

3.00 

12b. Performance Charts would be 
operationally useful 

1 = strongly agree 

5 = strongly disagree 

1.25 

2.00 


*p < .05 
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K.5 Flight Time Last 90 Days 


Table K-5 shows the mean response rates for survey questions as a function of flight time 
acquired in the last 90 days — less than 15 hrs. vs. greater than 15 hrs. None of the 
comparisons were significant, but again a few showed nearly significant differences; 
namely, 6b (p = .07), and 12a (p = .09). This indicates that pilots with less recent 
experience were more likely to agree that the interface was physically easy to use (6b). 
Furthermore, the results demonstrate that pilots with more recent experience agreed that 
the bookmark was useful. 


Table K-5. Mean Response Rates as a Function of Recent Flight Time 


Item 

Scoring Scale 

<15 

hrs. 

>15 

hrs. 

1. IE-NATOPS training organization 

1 = too loose 

2 = perfect 

3 = too regimented 

2.20 

2.00 

2. Amount of 

EE-NATOPS training 

1 = too little 

2 = sufficient 

3 = too much 

2.20 

2.00 

3. Amount of 

IE-NATOPS practice 

1 = too little 

2 = sufficient 

3 = too much 

2.20 

2.00 

4. Difficulty of IE-NATOPS training 
scenarios 

1 = too easy 

2 = sufficient 

3 = too complex 

■ 

2.00 

5. Confidence for in-flight use of IE- 
NATOPS 

1 = very confident 

5 = very uncertain 

a 

1.60 

6a. Paper NATOPS interface easy to 
manipulate 

1 = strongly agree 

5 = strongly disagree 

3.20 

2.40 

6b. IE-NATOPS interface easy to 
manipulate 

1 = strongly agree 

5 = strongly disagree 

1.20 

1.80 

7a. Always knew how to use paper 
NATOPS interface 

1 = strongly agree 

5 = strongly disagree 

2.80 

1.60 

7b. Always knew how to use IE-NATOPS 
interface 

1 = strongly agree 

5 = strongly disagree 

1.80 

2.00 

8a. Had no trouble reading & 
understanding paper NATOPS 

1 = strongly agree 

5 = strongly disagree 

1.20 

1.40 

8b. Had no trouble reading & 
understanding EE-NATOPS 

1 = strongly agree 

5 = strongly disagree 

1.60 

1.60 

9a. Had no trouble finding data with paper 
NATOPS 

1 = strongly agree 

5 = strongly disagree 

3.20 

1.80 

9b. Had no trouble finding data with paper 
NATOPS 

1 = strongly agree 

5 = strongly disagree 

2.00 

2.20 

10a. Easy to search multiple sections with 
paper NATOPS 

1 = strongly agree 

5 = strongly disagree 


2.20 
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10b. Easy to search multiple sections with 
IE-NATOPS 


11. For EE-NATOPS: 


11a. History function easy to use 


1 = strongly agree 
5 = strongly disagree 


1 = strongly agree 
5 = strongly disagree 




1 Id. Tab Feature in Checklists easy to use 


1 le. Search Feature in NATOPS easy to 
use 


1 If. Performance Charts easy to use 


12. For EE-NATOPS: 


1 = strongly agree 
5 = strongly disagree 


1 = strongly agree 
5 = strongly disagree 


1 = strongly agree 
5 = strongly disagree 





1 lb. Bookmark function easy to use 

1 = strongly agree 

5 = strongly disagree 





1 lc. Navigation Keys easy to use 

1 = strongly agree 

5 = strongly disagree 



2.1 




12a. Bookmark Function would be 
operationally useful 

1 = strongly agree 

5 = strongly disagree 

3.00 

2.00 

12b. Performance Charts would be 
operationally useful 

1 = strongly agree 

5 = strongly disagree 

1.75 

1.60 


K.6 Time in H-60F Simulator 

Table K-6 shows the mean response rates as a function of total time in the H-60F 
simulator — less than 100 hrs. vs. greater than 100 hrs. None of the comparisons were 
significant, nor did they approach significance in this category. 


Table K-6. Mean Response Rate as a Function of Total Time in H-60F Simulator 


Item 

Scoring Scale 



1. IE-NATOPS training organization 

1 = too loose 

2 = perfect 

3 = too regimented 

2.20 

2.00 

2. Amount of 

IE-NATOPS training 

1 = too little 

2 = sufficient 

3 = too much 

2.20 

2.00 

3. Amount of 

IE-NATOPS practice 

1 = too little 

2 = sufficient 

3 = too much 

2.20 

2.00 

4. Difficulty of IE-NATOPS training 
scenarios 

1 = too easy 

2 = sufficient 

3 = too complex 

2.20 

2.00 

5. Confidence for in-flight use of IE- 
NATOPS 

1 = very confident 

5 = very uncertain 

1.40 

1.80 

6a. Paper NATOPS interface easy to 
manipulate 

1 = strongly agree 

5 = strongly disagree 

3.20 

2.40 
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6b. IE-NATOPS interface easy to 
manipulate 

1 = strongly agree 

5 = strongly disagree 

1.40 

1.60 

7a. Always knew how to use paper 
NATOPS interface 

1 = strongly agree 

5 = strongly disagree 

2.80 

1.60 

7b. Always knew how to use IE-NATOPS 
interface 

1 = strongly agree 

5 = strongly disagree 

1.80 

2.00 

8a. Had no trouble reading & 
understanding paper NATOPS 

1 = strongly agree 

5 = strongly disagree 

1.40 

1.20 

8b. Had no trouble reading & 
understanding EE-NATOPS 

1 = strongly agree 

5 = strongly disagree 

1.80 

1.40 

9a. Had no trouble finding data with paper 
NATOPS 

1 = strongly agree 

5 = strongly disagree 

2.80 

2.20 

9b. Had no trouble finding data with paper 
NATOPS 

1 = strongly agree 

5 = strongly disagree 

2.00 

2.20 

10a. Easy to search multiple sections with 
paper NATOPS 

1 = strongly agree 

5 = strongly disagree 

2.00 


10b. Easy to search multiple sections with 
IE-NATOPS 

1 = strongly agree 

5 = strongly disagree 

1.40 

mi 

11. For IE-NATOPS: 




1 la. History function easy to use 

1 = strongly agree 

5 = strongly disagree 

2.20 

2.20 

lib. Bookmark function easy to use 

1 = strongly agree 

5 = strongly disagree 


2.20 

1 lc. Navigation Keys easy to use 

1 = strongly agree 

5 = strongly disagree 

1.60 

1.80 

lid. Tab Feature in Checklists easy to use 

1 = strongly agree 

5 = strongly disagree 

1.40 

1.40 

1 le. Search Feature in NATOPS easy to 
use 

1 = strongly agree 

5 = strongly disagree 

1.80 

1.60 

1 If. Performance Charts easy to use 

1 = strongly agree 

5 = strongly disagree 

1.60 

1.40 

12. For IE-NATOPS: 




12a. Bookmark Function would be 
operationally useful 

1 = strongly agree 

5 = strongly disagree 

2.75 


12b. Performance Charts would be 
operationally useful 

1 = strongly agree 

5 = strongly disagree 

1.75 

pm 


| K.7 Time in Rotary Wing 

I Table K-7 shows the mean response rates on the survey questions as a function of total 

flight time in rotary wing — less than 1500 hrs. vs. greater than 1500 hrs. None of these 
comparisons were significant; however, one of the comparisons approached significance 

| (p = .07), and that was for the item associated with the benchmark function and its ease of 
use. In this case, pilots with less than 1500 hrs. in rotary wing rated this function as 
easier to use in comparison to those pilots with over 1500 hrs. 

I 
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